How to define a DaemonSet to create slightly different pods?

[Dongwon Kim]

How can I define a single definition of DaemonSet to create slightly different pods from it?

What I mean by different pods is that pods created from different commands.

I currently make two different DaemonSet objects to create two different kinds of pods.

Let me tell you the details.

I've been modifying kube-flannel.yml, especially the definition of DaemonSet.

I want to add "--iface=<ethernet interface>" to flanneld [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr"]

and I want to use two different ethernet interfaces based on node label. 

I have three different kinds of nodes:

1) a master node with two ethernet interfaces: eth0 for internal network and eth3 for external network 

2) slave nodes with one ethernet interfaces: eth0 for internal network

3) a newly added node with one ethernet interfaces: p2p1 for internal network 

When I had 1) and 2), I was okay with the following command:

[ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=--eth0"] 

I have to add --iface option because otherwise flanneld on the master node automatically detects eth3 other than eth0, 

which is the reason I manually add the iface option.

Now that I have 3),

I have to make two DaemonSet definitions with different commands:

[ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=--eth0"]  for 1) and 2)

, [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=--p2p1"] for 3) 

Label selector is used to tell which DaemonSet is used for a node. 

The entire definition is attached at the end. 

Is there a more elegant way of creating slightly different pods from a single DaemonSet definition?

It would be nice to have only a single definition with a placeholder which can be substituted by the value of a specific label. 

Otherwise I can do this by setting flannel in a different way?

It would be nice if we can give a list of ethernet interfaces to --iface option but my colleague found that only one interface/ip can be given to --iface option.

Thanks.

---

apiVersion: v1

kind: ServiceAccount

metadata:

  name: flannel

---

kind: ConfigMap

apiVersion: v1

metadata:

  name: kube-flannel-cfg

  labels:

    tier: node

    app: flannel

data:

  cni-conf.json: |

    {

      "name": "cbr0",

      "type": "flannel",

      "delegate": {

        "isDefaultGateway": true

      }

    }

  net-conf.json: |

    {

      "Network": "10.244.0.0/16",

      "Backend": {

        "Type": "vxlan"

      }

    }

--- 

apiVersion: extensions/v1beta1

kind: DaemonSet

metadata:

  name: kube-flannel-ds

  labels:

    tier: node

    app: flannel

    iface: eth0

spec:

  template:

    metadata:

      labels:

        tier: node

        app: flannel

        iface: eth0

    spec:

      hostNetwork: true

      nodeSelector:

        beta.kubernetes.io/arch: amd64

        iface: eth0

      serviceAccountName: flannel

      containers:

      - name: kube-flannel

        image: quay.io/coreos/flannel:v0.7.0-amd64

        command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=eth0" ]

        securityContext:

          privileged: true

        env:

        - name: POD_NAME

          valueFrom:

            fieldRef:

              fieldPath: metadata.name

        - name: POD_NAMESPACE

          valueFrom:

            fieldRef:

              fieldPath: metadata.namespace

        volumeMounts:

        - name: run

          mountPath: /run

        - name: flannel-cfg

          mountPath: /etc/kube-flannel/

      - name: install-cni

        image: quay.io/coreos/flannel:v0.7.0-amd64

        command: [ "/bin/sh", "-c", "set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf; while true; do sleep 3600; done" ]

        volumeMounts:

        - name: cni

          mountPath: /etc/cni/net.d

        - name: flannel-cfg

          mountPath: /etc/kube-flannel/

      volumes:

        - name: run

          hostPath:

            path: /run

        - name: cni

          hostPath:

            path: /etc/cni/net.d

        - name: flannel-cfg

          configMap:

            name: kube-flannel-cfg

---

apiVersion: extensions/v1beta1

kind: DaemonSet

metadata:

  name: kube-gpu-flannel-ds

  labels:

    tier: node

    app: flannel

    iface: p2p1

spec:

  template:

    metadata:

      labels:

        tier: node

        app: flannel

        iface: p2p1

    spec:

      hostNetwork: true

      nodeSelector:

        beta.kubernetes.io/arch: amd64

        iface: p2p1

      serviceAccountName: flannel

      containers:

      - name: kube-flannel

        image: quay.io/coreos/flannel:v0.7.0-amd64

        command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=p2p1" ]

        securityContext:

          privileged: true

        env:

        - name: POD_NAME

          valueFrom:

            fieldRef:

              fieldPath: metadata.name

        - name: POD_NAMESPACE

          valueFrom:

            fieldRef:

              fieldPath: metadata.namespace

        volumeMounts:

        - name: run

          mountPath: /run

        - name: flannel-cfg

          mountPath: /etc/kube-flannel/

      - name: install-cni

        image: quay.io/coreos/flannel:v0.7.0-amd64

        command: [ "/bin/sh", "-c", "set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf; while true; do sleep 3600; done" ]

        volumeMounts:

        - name: cni

          mountPath: /etc/cni/net.d

        - name: flannel-cfg

          mountPath: /etc/kube-flannel/

      volumes:

        - name: run

          hostPath:

            path: /run

        - name: cni

          hostPath:

            path: /etc/cni/net.d

        - name: flannel-cfg

          configMap:

            name: kube-flannel-cfg

[Rodrigo Campos]

Don't know about flannel if a better way in that layer can be done.

But you can do deployments with hostPort (so no two are on the same node) and adjust the number of replicas as you change the number of nodes.

the other way I see is using daemonSet but having the logic on the command to run inside the container (like a script that checks which command to run based on what you need and "exec" that)

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

[Brandon Philips]

adding Tom and Casey for Flannel help

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.

[Brandon Philips]

Dongwon-

One approach would be configuring your systems so that the interface you want to use is named the same on all machines. You can do this by putting down a UDEV rule: http://unix.stackexchange.com/questions/91085/udev-renaming-my-network-interface

So, p2p1 becomes ex0 and eth0 becomes ex0, etc

Hope that helps.

Brandon

[김동원]

Hi Rodrgio :-)

But you can do deployments with hostPort (so no two are on the same node) and adjust the number of replicas as you change the number of nodes.

This way seems the same as the DaemonSet approach in that I have to specify two Deployments each with different templates to create containers.

the other way I see is using daemonSet but having the logic on the command to run inside the container (like a script that checks which command to run based on what you need and "exec" that)

I want to identify an ethernet interface on the *host* when launching a container, which cannot be identified inside a container.

I could attach to the flannel container a "hostPath" volume which specifies a system-related location on the *host* so that a script running inside a container can identify the ethernet interface from the location.

This approach doesn't look desirable to me, though.

Correct me if I didn't understand your intention :-)

Thanks

- Dongwon

2017. 2. 23. 오전 3:40, Rodrigo Campos <rodr...@gmail.com> 작성:

You received this message because you are subscribed to a topic in the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-users/F6GvfHoX7V4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.

[김동원]

Oops, I just found that kube-flannel containers can see the ethernet interfaces on the host!!

Ignore the previous one.

As you said, I could add a script to identify an ethernet interface.

Thanks!

2017. 2. 23. 오전 10:56, 김동원 <eastc...@gmail.com> 작성:

[김동원]

Hi Brandon :-)

One approach would be configuring your systems so that the interface you want to use is named the same on all machines. You can do this by putting down a UDEV rule: http://unix.stackexchange.com/questions/91085/udev-renaming-my-network-interface

So, p2p1 becomes ex0 and eth0 becomes ex0, etc

Renaming network interfaces looks the simplest solution because I just need to use the same value (e.g. ex0) for --iface.

Nevertheless, scripting must be preferred to create different pods from a DaemonSet

as making all nodes homogenous could not make sense for some other cases.

- Dongwon

2017. 2. 23. 오전 5:00, Brandon Philips <brandon...@coreos.com> 작성:

You received this message because you are subscribed to a topic in the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-users/F6GvfHoX7V4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-use...@googlegroups.com.

[Tom Denham]

The only thing I can think of involves enhancing flannel. But I think this would be a good thing to do so I raised an issue to track it

https://github.com/coreos/flannel/issues/620

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

[김동원]

Hi Tom,

Nice to see the issue.

Hope it will soon be available!

Thanks,

- Dongwon

2017. 2. 24. 오전 10:59, Tom Denham <t...@tigera.io> 작성:

You received this message because you are subscribed to a topic in the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-users/F6GvfHoX7V4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.