FQDN's for pods?

[David Rosenstrauch]

Is it possible to make Kubernetes assign fully-qualified domain names to
pods at launch? I know Docker supports this using the "-h" flag (e.g.,
"docker run -h host1234.ourdomain.com ...") but I don't see a
corresponding way to trigger that functionality in containers launched
by k8s.

We have a requirement for this, as we have piece of software for which
we have a site license that we want to containerize. However the s/w
does the site license verification by checking the domain suffix of the
host it's running on. (I.e., using "hostname -f") So if we can't get
our pods and services to use the domain name suffix, we won't be able to
run the app inside a container.

I've searched a bit, but don't see a definitive solution to the issue.
Any suggestions much appreciated!

Thanks,

DR

[Tim Hockin]

https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#a-records-and-hostname-based-on-pods-hostname-and-subdomain-fields

?

> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kubernetes-use...@googlegroups.com.
> To post to this group, send email to kubernet...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kubernetes-users.
> For more options, visit https://groups.google.com/d/optout.

[Brandon Philips]

That won't do what he wants, I don't think.

$ kubectl run -i -t busybox --image=busybox --restart=Never -n team-tectonic --overrides='{ "apiVersion": "v1", "spec": {"hostname": "hello", "subdomain": "example"}}'

If you don't see a command prompt, try pressing enter.

/ # hostname -f

hello.example.team-tectonic.svc.cluster.local

/ #

Essentially he wants hostname and subdomain to combine and set the FQDN. Not just the start of the FQDN. E.g. hello.example

Brandon

[Matthias Rampke]

If it's checking the domain suffix, everything should work if you set the cluster domain to a subdomain of yours instead of cluster.local – then the name will be of the form <pod>.<namespace>.pod.<subdomain>.<domain>, no? We use this in all our clusters, but we make a custom distribution, so how to do this will depend on how you build the cluster(s).

Or, dirty but simple, can you manipulate the pod hostname and domain from inside the pod? Call `hostname` and mess with /etc/hosts?

/MR

[Tim Hockin]

We don't have any mechanism to set the FQDN, other than this, for now.

[David Rosenstrauch]

On 2017-09-05 5:04 pm, Brandon Philips wrote:
> That won't do what he wants, I don't think.
>
> $ kubectl run -i -t busybox --image=busybox --restart=Never -n
> team-tectonic --overrides='{ "apiVersion": "v1", "spec": {"hostname":
> "hello", "subdomain": "example"}}'
> If you don't see a command prompt, try pressing enter.
> / # hostname -f
> hello.example.team-tectonic.svc.cluster.local
> / #
>
> Essentially he wants hostname and subdomain to combine and set the
> FQDN. Not just the start of the FQDN. E.g. hello.example
>
> Brandon

What I'm looking to see from "hostname -f" is actually more like this:

<some-pod-name>.<ourdomain>.com

If the fqdn ends with "cluster.local", the license check will fail.

It doesn't look like that can be achieved from what you're describing
here, since the subdomain ("example") appears in the middle of the FQDN,
rather than as the suffix.

DR

[David Rosenstrauch]

On 2017-09-05 5:39 pm, Matthias Rampke wrote:
> If it's checking the domain suffix, everything should work if you set
> the cluster domain to a subdomain of yours instead of cluster.local
> – then the name will be of the form
> <pod>.<namespace>.pod.<subdomain>.<domain>, no? We use this in all our
> clusters, but we make a custom distribution, so how to do this will
> depend on how you build the cluster(s).
>
> Or, dirty but simple, can you manipulate the pod hostname and domain
> from inside the pod? Call `hostname` and mess with /etc/hosts?
>
> /MR

That sounds like more along the lines of what I want. How do I go about
setting the cluster domain then?

Thanks,

DR

[Tim Hockin]

We do not have a mechanism to express what you want to express, then.
You control the cluster suffix and the subdomain, and the pod name,
but even with all of those in play, the hostname comes out as
`<pod>.<subdomain>.svc.<suffix>`, I am pretty sure. I am open to
proposals on how to allow what you want.

[Quinn Comendant]

Perhaps use a wrapper for hostname that returns a simulated hostname if called from your special program:

#!/bin/bash
if [[ $(ps -o comm= $PPID) == '/your/app/here' ]]; then
echo "imitation.hostname.ourdomain.com"
else
/bin/hostname "$@"
fi

[Matthias Rampke]

This is set via the `--cluster-domain` flag on the kubelet, as well as in the kubedns deployment.

/MR

[David Rosenstrauch]

On 2017-09-05 6:19 pm, 'Tim Hockin' via Kubernetes user discussion and

Q&A wrote:
> We do not have a mechanism to express what you want to express, then.
> You control the cluster suffix and the subdomain, and the pod name,
> but even with all of those in play, the hostname comes out as
> `<pod>.<subdomain>.svc.<suffix>`, I am pretty sure. I am open to
> proposals on how to allow what you want.

Being able to set the suffix would suffice for my purposes. (I think
the license check just verifies that the fqdn ends with our domain.)

But I didn't see any k8s option to set that in such a way that calling
"hostname -f" from inside a pod would return a string ending with that
suffix. Is there a setting I'm missing?

Thanks,

DR

[David Rosenstrauch]

Wouldn't work. It's a 3rd party (closed source, binary only) software
that we're using, and that's what's calling hostname -f.

I suppose we could move /usr/bin/hostname out of the way, and run our
own, but that's rather a hack. (And I'm not even sure if the 3rd party
software is specifically using the hostname binary to get the fqdn.)

Thanks,

DR

[David Rosenstrauch]

I believe I tried that. IIRC, that does successfully result in dns
lookups returning fqdn's containing our domain suffix.

But what I'm trying to accomplish isn't DNS-based. Rather, I'm trying
to make "hostname -f" in a pod return a fqdn that contains our domain
suffix. IIUC, "hostname -f" gets its information from the /etc/hostname
file. So I think what that means is that what I actually want to happen
is to make k8s create a /etc/hostname file inside of each pod that
contains fqdn. Not sure how to do that. (Or if it's even possible.)

Thanks,

DR

[Rodrigo Campos]

As Matthias said, you can use the --cluster-domain if you want to. That is usually set at installation.

That may be the simplest thing, and maybe fits better in your organization using your domain for the cluster.

But if the license check is doing that, I bet that you can use LD_PRELOAD to hack it, and use strace to see what calls it's really using.

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

[David Rosenstrauch]

On 2017-09-05 6:19 pm, 'Tim Hockin' via Kubernetes user discussion and

Q&A wrote:
> We do not have a mechanism to express what you want to express, then.
> You control the cluster suffix and the subdomain, and the pod name,
> but even with all of those in play, the hostname comes out as
> `<pod>.<subdomain>.svc.<suffix>`, I am pretty sure. I am open to
> proposals on how to allow what you want.

Docker allows us to launch containers this way. For example, I can
perform the following:

sudo docker run -d -h "$(openssl rand -hex 8).ourdomain.com" -p
33334:33333 the-container-image

And then the container that gets launched will successfully report back
a fully-qualified hostname that includes the domain name when I issue a
hostname -f.


In answer to your comment about proposals on how to allow what I want, I
guess conceptually I'd think the following:

Kubernetes is already setting the hostname for each pod that gets
launched - usually in the form of
<service-name>-<replicaset-id>-<some-unique-hash>. I guess then what
would make sense to me is that this be changed to be
<service-name>-<replicaset-id>-<some-unique-hash>.<cluster-domain>. In
most cases, then, your fully-qualified pod hostname would look like
abc-1234567-12345.cluster.local. But if you chose to override the
cluster domain on your cluster, you could then easily make it change to
abc-1234567-12345.yourdomain.com.

Any reason why this couldn't work, or shouldn't be done? (I.e., any
particular reason why a pod hostname should not always be set to a
fqdn?)

Thanks,

DR

[Matthias Rampke]

TL;DR when you set the cluster domain, this should Just Work™ in Kubernetes 1.7+ but not before

David – what Kubernete version are you running? I just went down a rabbit hole because our /etc/hosts did not look like this code[0] suggests it should. Turns out, there was a bug before 1.7[1] that causes /etc/hosts to not actually be written at the right times.

From my understanding  the `hostname -f` mechanism[2][3] ultimately takes its information from the line like

10.3.4.5 podname podname.cluster-domain

but even though we set a cluster domain, I only got the short name. Issue 44473 suggests that this is fixed in 1.7 but I have no such cluster to test it against.

/MR

[0] https://github.com/kubernetes/kubernetes/blob/7be29bd9b6913e2f39b44a72a5f46e6f33244410/pkg/kubelet/kubelet_pods.go#L300-L304

[1] https://github.com/kubernetes/kubernetes/issues/44473

[2] https://linux.die.net/man/1/hostname

[3] https://linux.die.net/man/3/gethostbyname

[David Rosenstrauch]

On 2017-09-06 2:36 pm, Matthias Rampke wrote:
> TL;DR when you set the cluster domain, this should Just Work™ in
> Kubernetes 1.7+ but not before

That's good news! I'll start to look into us upgrading to a newer
version.

> David – what Kubernete version are you running?

We're running v1.5.2. (The version that ships with Centos 7.)


Thanks very much for looking into this!

DR

[David Rosenstrauch]

On 2017-09-06 2:42 pm, David Rosenstrauch wrote:
> On 2017-09-06 2:36 pm, Matthias Rampke wrote:
>> TL;DR when you set the cluster domain, this should Just Work™ in
>> Kubernetes 1.7+ but not before
>
> That's good news! I'll start to look into us upgrading to a newer
> version.

Hmmm ... some bad news. I built a new cluster using k8s v1.7.5, and I'm
still seeing the same issue:

[centos@ip-172-31-95-205 ~]$ kubectl exec -it nginx-4217019353-9w7f5
/bin/bash
root@nginx-4217019353-9w7f5:/# hostname -f
nginx-4217019353-9w7f5
root@nginx-4217019353-9w7f5:/# cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.240.101.2 nginx-4217019353-9w7f5

Any idea what might be the problem / how I might go about fixing?

Thanks,

DR

[Tim Hockin]

We don't set the FQDN at all unless you specify the `subdomain` field
in Pod.spec. That could be a bug, but the assumption is that the FQDN
is "about" DNS lookup-ability. We could have that conversation,
though.

[dar...@darose.net]

Ah, there we are.  That worked!  (Even on the v1.5.2 that ships with Centos 7.)

I set the subdomain to "pod" and now I see the following:

[centos@ip-172-31-93-54 ~]$ kubectl exec -it custom-django-app-1796634124-7jmd4 /bin/bash
root@custom-django-app-1796634124-7jmd4:/django-apps/challenge1# hostname -f
custom-django-app-1796634124-7jmd4.pod.default.svc.<ourdomain>

Thanks much for the help!

DR

----- Original Message -----

From:

kubernet...@googlegroups.com

To:

"Kubernetes user discussion and Q&A" <kubernet...@googlegroups.com>

Cc:

Sent:

Fri, 8 Sep 2017 09:43:37 -0700

Subject:

Re: [kubernetes-users] FQDN's for pods?