Multus - Bridge Mode | cni0 not cleaned automatically

[Rahul Gupta]

Dear Community,

I would like to have your comments on the specific scenario.

Multus - Bridge Mode - Networkattachmentdefinition

I've a POD with multiple networks on one of the worker nodes which is deployed using a statefulset.  One of the secondary network is used in the bridge mode. When the pod is created on the worker, cni0 bridge is created and veth interface is connected to that cni0.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  annotations:
    meta.helm.sh/release-name: test
    meta.helm.sh/release-namespace: devtest
  labels:
    app: testapp
    app.kubernetes.io/managed-by: Helm
  name: test-network-3
  namespace: devtest
spec:
  config: '{ "plugins": [ { "cniVersion": "0.3.0", "type": "bridge", "ipam": { "type":
    "host-local-test" } }, { "capabilities": { "mac": true }, "type": "tuning" } ]
    }'

If now, I delete this pod and force it to be created on another worked. It gets created on new worker node but cni0 bridge is not cleaned up from the previous worker and cni0 remains in down state.

Has anybody seen this issue ? Is it normal to have a cni0 as down where unused? 

Also, may I know which component is responsible for creating and deleting the bridge from workers ?

Thanks,

Rahul

[Dan Winship]

Multus isn't part of core Kubernetes, but I'm not sure if there are any
Multus email lists?

At any rate, while "high level" network plugins tend to have daemons
that keep track of all of the pods on a node that are making use of
them, "low level" CNI plugins (like "bridge") don't have that, so when
you delete the original pod, the CNI plugin doesn't know for sure that
you're done with the bridge (and that no one else is planning to use
it), so it doesn't remove it. That might be annoying, but it's not
considered a bug in the bridge plugin. (Though perhaps there could be a
config option to change that?)

-- Dan

On 7/3/23 09:06, Rahul Gupta wrote:
> Dear Community,
>
> I would like to have your comments on the specific scenario.
>

> *Multus - Bridge Mode - Networkattachmentdefinition*
> *
> *

> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to kubernetes-sig-ne...@googlegroups.com
> <mailto:kubernetes-sig-ne...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kubernetes-sig-network/c1e7cb42-c62b-4cc4-8ee0-500f64cb553fn%40googlegroups.com <https://groups.google.com/d/msgid/kubernetes-sig-network/c1e7cb42-c62b-4cc4-8ee0-500f64cb553fn%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Rahul Gupta]

Thanks Dan for the response. I tried to look for the config option but couldn't find it in the bridge plugins documentation and not in the multus plugin as well.

This is definitely annoying especially when you've alerting enabled for the Interface Outage on the workers and cni0 Down pops-up in the alerts. :)

/Rahul

[Casey Callendrello]

We'd been discussing in the CNI project about better network lifecycle events, e.g. INITIALIZE and DEINITIALIZE. We've decided at this point to drop them from CNI v1.1, since the use-case is not particularly compelling. Specifically, the lifecycle of a network within the entire Kubernetes / CRI / CNI ecosystem doesn't currently have a good point in which to declare "this network configuration is done, please delete it."

We can revisit this for v1.2 if there is enough desire from the community. CNI v1.1 is already looking to be a big release, so I'd like to keep more things out of it for now.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-network/CAEvs4-5a0dx-r8Rg%3Djz6cwVNGvO-LRE2ahPtyv%3DWvV1_yfcpEw%40mail.gmail.com.