Bridge CNI across hosts

[Zvi Cahana]

Hi all.

I'm experimenting with a setup of multus+bridge cni+bandwidth metaplugin over OCP4.

I'd like to understand how should I connect the bridge to achieve connectivity for pods across different nodes.

Here's the CNI config I've used:

{

  "cniVersion": "0.4.0",

  "name": "zvic--nss2",

  "type": "bridge",

  "bridge": "br1",

  "isGateway": true,

  "isDefaultGateway": true,

  "forceAddress": true,

  "ipMasq": true,

  "mtu": 0,

  "hairpinMode": true,

  "promiscMode": false,

  "vlan": 0,

  "ipam": {

"type": "whereabouts",

"name": "zvic--nss2",

"datastore": "kubernetes",

"range": "100.10.5.0/24",

"kubernetes": {}

  }

}

on the host side, I executed:

ip link set ens192 master br1

where ens192 is the primary interface for the node which is connected to other nodes, and br1 is the bridge referenced by the above CNI config.

Shortly after doing this, the nodes have lost their connectivity via ens192 , so I figured I bricked their network via the above.

So, what's the right way to configure the bridge and connect it to the outside world?

Thanks in advance,

Zvi

[Dan Kenigsberg]

I suspect that by piping L2 traffic into b1 you deprive it from ens192's ip stack. You have to set the op address on the bridge.

Shameless plug: use https://github.com/nmstate/kubernetes-nmstate/blob/master/docs/examples/linux-bridge.yaml to easily apply even more complex node network configurations to your cluster.

Regards, Dan

Thanks in advance,

Zvi

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-network/769fdacd-64db-4e78-8753-14a9beab5fccn%40googlegroups.com.

[Dan Winship]

On 10/17/20 4:22 PM, Zvi Cahana wrote:
> Hi all.
>
> I'm experimenting with a setup of multus+bridge cni+bandwidth metaplugin
> over OCP4.
> I'd like to understand how should I connect the bridge to achieve
> connectivity for pods across different nodes.

The bridge plugin isn't designed to do multi-node networks. (It's
intended for something like docker/podman where you're only setting up
containers on a single host.)

You might be able to make it work with multiple hosts, but it's going to
involve weird hacks. (There's not just some simple configuration option
that you missed or something like that.)

-- Dan

>
> Here's the CNI config I've used:

> *{*
> *  "cniVersion": "0.4.0",*
> *  "name": "zvic--nss2",*
> *  "type": "bridge",*
> *  "bridge": "br1",*
> *  "isGateway": true,*
> *  "isDefaultGateway": true,*
> *  "forceAddress": true,*
> *  "ipMasq": true,*
> *  "mtu": 0,*
> *  "hairpinMode": true,*
> *  "promiscMode": false,*
> *  "vlan": 0,*
> *  "ipam": {*
> *"type": "whereabouts",*
> *"name": "zvic--nss2",*
> *"datastore": "kubernetes",*
> *"range": "100.10.5.0/24",*
> *"kubernetes": {}*
> *  }*
> *}*

>
>
> on the host side, I executed:

> *ip link set ens192 master br1*
>
> where *ens192 *is the primary interface for the node which is connected
> to other nodes, and *br1 *is the bridge referenced by the above CNI config.

>
> Shortly after doing this, the nodes have lost their connectivity

> via *ens192 *, so I figured I bricked their network via the above.

>
> So, what's the right way to configure the bridge and connect it to the
> outside world?
>
> Thanks in advance,
> Zvi
>

> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to kubernetes-sig-ne...@googlegroups.com

> <mailto:kubernetes-sig-ne...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kubernetes-sig-network/769fdacd-64db-4e78-8753-14a9beab5fccn%40googlegroups.com

> <https://groups.google.com/d/msgid/kubernetes-sig-network/769fdacd-64db-4e78-8753-14a9beab5fccn%40googlegroups.com?utm_medium=email&utm_source=footer>.