NMPolicy - a cmd/lib tool to generate dynamic NMState configurations

Alona Paz

unread,

Dec 16, 2021, 4:24:38 AM12/16/21

to kubevirt-dev, nmstat...@lists.fedorahosted.org

Hi all,

The kubevirt networking team is working on a new cmd/lib tool to generate dynamic NMState configurations.

When networking configuration for a cluster is needed and all the details are common between the nodes in the cluster a NMState yaml configuration is enough.

Problems arise when some of the network configuration details are different between nodes and depend on the current node network state.

For that a different NMState yaml configuration needs to be generated per node and that's not convenient for big clusters and also at scale up scenarios.

The NMPolicy goal is to solve this problem.

Given a node network state and a network configuration policy (common to the cluster), the NMPolicy tool will generate a node specific desired network state.

NMPolicy was already integrated to kubernetes-nmstate and a usage example can be found here.

CLI Usage example -

Simulated --help output

./nmpolicyctl -h

This tool helps you to generate dynamic NMState configurations.

The generated NMState configuration is written to STDOUT.

Usage of ./nmpolicyctl:

Commands:

gen Generates NMState by policy filename

Flags:

-s, --current-state=current-state.yaml: input file path to current NMState. If not specified, stdin is used.

-c, --captured-state-output=captured-state-output.yaml: output file path to the emitted captured states. If not specified, ~/.cache/nmpolicy/cache.yaml will be used.

Creating a new bridge with the base interface as its port. The bridge will get the mac address of the base interface.

# echo """

capture:

default-gw: routes.running.destination=="0.0.0.0/0"

base-iface: interfaces.name==capture.default-gw.routes.running.0.next-hop-interface

desiredState:

interfaces:

- name: br1

description: Linux bridge with base interface as a port

type: linux-bridge

state: up

mac-address: "{{ capture.base-iface.interfaces.0.mac-address }}"

ipv4:

dhcp: true

enabled: true

bridge:

options:

stp:

enabled: false

port:

- name: "{{ capture.base-iface.interfaces.0.name }}"

""" > policy.yaml

# nmstatectl show | nmpolicyctl gen policy.yaml | nmstatectl apply

Output of - nmstatectl show | nmpolicyctl gen policy.yaml

~/.cache/nmpolicy/cache.yaml (the default captured states output file)

base-iface:

metaInfo:

time: "2021-12-15T13:45:40Z"

state:

interfaces:

- accept-all-mac-addresses: false

ethernet:

auto-negotiation: false

ethtool:

feature:

rx-gro: true

rx-gro-list: false

rx-udp-gro-forwarding: false

tx-checksum-ip-generic: true

tx-generic-segmentation: true

tx-nocache-copy: false

tx-tcp-ecn-segmentation: true

tx-tcp-mangleid-segmentation: false

tx-tcp-segmentation: true

tx-tcp6-segmentation: true

ring:

rx: 256

tx: 256

ipv4:

address:

- ip: 192.168.66.101

prefix-length: 24

auto-dns: true

auto-gateway: true

auto-route-table-id: 0

auto-routes: true

dhcp: true

enabled: true

ipv6:

address:

- ip: fd00::101

prefix-length: 128

- ip: fe80::2d55:7c37:8090:7261

prefix-length: 64

auto-dns: true

auto-gateway: true

auto-route-table-id: 0

auto-routes: true

autoconf: true

dhcp: true

enabled: true

lldp:

enabled: false

mac-address: 52:55:00:D1:55:01

mtu: 1500

name: eth0

state: up

type: ethernet

default-gw:

metaInfo:

time: "2021-12-15T13:45:40Z"

state:

routes:

running:

- destination: 0.0.0.0/0

metric: 100

next-hop-address: 192.168.66.2

next-hop-interface: eth0

table-id: 254

Stdout (the desired state to be applied by nmstatectl)

interfaces:

- bridge:

options:

stp:

enabled: false

port:

- name: eth0

vlan:

mode: trunk

trunk-tags:

- id-range:

max: 4094

min: 2

description: Linux bridge with base interface as a port

ipv4:

dhcp: true

enabled: true

mac-address: 52:55:00:D1:55:01

name: br1

state: up

type: linux-bridge

For more details, please review the design doc and share your feedback about it.

Thanks,

Alona.

Dan Kenigsberg

unread,

Dec 17, 2021, 9:58:35 AM12/17/21

to Alona Paz, kubevirt-dev, nmstat...@lists.fedorahosted.org

Thanks for sharing your plan.

I would like to create a Linux bond of all the NICs of my machine, create a Linux bridge on top of it, and copy my original mac/IP/routing to that bridge. Can you share a policy that would let me express this desired state?

Regards,

Dan.

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAJOMPacxNke2pJnjJ1PKxz0f%3Dmm%3DrNpnWqZwAMtNXf7f0HXL_Q%40mail.gmail.com.

Felix Enrique Llorente Pastora

unread,

Dec 17, 2021, 10:25:52 AM12/17/21

to Dan Kenigsberg, Alona Paz, kubevirt-dev, nmstat...@lists.fedorahosted.org

On Fri, Dec 17, 2021 at 3:58 PM Dan Kenigsberg <dan...@redhat.com> wrote:

Thanks for sharing your plan.

I would like to create a Linux bond of all the NICs of my machine, create a Linux bridge on top of it, and copy my original mac/IP/routing to that bridge. Can you share a policy that would let me express this desired state?

You need to know the number of NICs, for example for three of them this would be it.

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: bra1
spec:
capture:
default-gw: routes.running.destination=="0.0.0.0/0"
primary-iface: interfaces.name==capture.default-gw.routes.running.0.next-hop-interface
ethernet-ifaces: interface.type==ethernet
desiredState:
interfaces:
- name: bond1
type: bond
state: up
link-aggregation:
mode: balance-rr
options:
miimon: '140'
port:
- name: "{{ capture.ethernet-ifaces.interfaces.0.name }}"
- name: "{{ capture.ethernet-ifaces.interfaces.1.name }}"
- name: "{{ capture.ethernet-ifaces.interfaces.2.name }}"

- name: br1
description: Linux bridge with base interface as a port
type: linux-bridge
state: up

mac-address: "{{ capture.primary-iface.interfaces.0.mac-address }}"
ipv4: "{{ capture.primary-iface.interfaces.0.ipv4 }}"

bridge:
options:
stp:
enabled: false
port:

- bondall

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAHOEP56e4zhHPtP8rZp%3D%3DZ_zf_XyTb-hOB%2BvVOiwMX%3DniqRNrg%40mail.gmail.com.

--

Quique Llorente

CNV networking Senior Software Engineer

Red Hat EMEA

ello...@redhat.com

@RedHat Red Hat Red Hat

Dan Kenigsberg

unread,

Dec 19, 2021, 3:42:24 AM12/19/21

to Felix Enrique Llorente Pastora, Alona Paz, kubevirt-dev, nmstat...@lists.fedorahosted.org

On Fri, Dec 17, 2021 at 5:25 PM Felix Enrique Llorente Pastora <ello...@redhat.com> wrote:

On Fri, Dec 17, 2021 at 3:58 PM Dan Kenigsberg <dan...@redhat.com> wrote:
Thanks for sharing your plan.

I would like to create a Linux bond of all the NICs of my machine, create a Linux bridge on top of it, and copy my original mac/IP/routing to that bridge. Can you share a policy that would let me express this desired state?

You need to know the number of NICs, for example for three of them this would be it.

Thanks. Having a Python-like `0:-1` slice index to select a sublist could be even cooler.

More importantly, I would like to be able to select all interfaces which have vlan 204 enabled on their switch port, as reported by lldp. What would the capture look like in this case?

Is the suggested language expressive enough to capture them? Here's my (redacted) nmstate show for one such interface. I find the LLDP portion there a bit confusing.

- ethernet:
auto-negotiation: true
duplex: full
speed: 10000
ipv4:
address: []

auto-dns: true
auto-gateway: true
auto-route-table-id: 0
auto-routes: true
dhcp: true
enabled: true
ipv6:

address: []
autoconf: false
dhcp: false
enabled: false
lldp:
enabled: true
neighbors:
- - system-name: sw01-access-f1.example.com
type: 5
- system-description: 'Juniper Networks, Inc'
type: 6
- system-capabilities:
- MAC Bridge component
- Router
type: 7
- _description: MAC address
chassis-id-type: 4
type: 1
- _description: Interface name
port-id: xe-0/0/17
port-id-type: 5
type: 2
- ieee-802-1-vlans:
- name: vlan-204
vid: 204
- name: vlan-205
vid: 205
subtype: 3
type: 127
- ieee-802-3-mac-phy-conf:
autoneg: false
operational-mau-type: 0
pmd-autoneg-cap: 32768
oui: 00:12:0f
subtype: 1
type: 127
- management-addresses:
address-subtype: ipv4
interface-number: 0
interface-number-subtype: 1
type: 8
- ieee-802-3-max-frame-size: 9216
oui: 00:12:0f
subtype: 4
type: 127
mtu: 1500
name: ens1f0
state: up
type: ethernet

I suspect you need `bond1` here?

Alona Paz

unread,

Dec 19, 2021, 4:20:28 AM12/19/21

to Dan Kenigsberg, Felix Enrique Llorente Pastora, kubevirt-dev, nmstat...@lists.fedorahosted.org

On Sun, Dec 19, 2021 at 10:42 AM Dan Kenigsberg <dan...@redhat.com> wrote:

On Fri, Dec 17, 2021 at 5:25 PM Felix Enrique Llorente Pastora <ello...@redhat.com> wrote:

On Fri, Dec 17, 2021 at 3:58 PM Dan Kenigsberg <dan...@redhat.com> wrote:
Thanks for sharing your plan.

I would like to create a Linux bond of all the NICs of my machine, create a Linux bridge on top of it, and copy my original mac/IP/routing to that bridge. Can you share a policy that would let me express this desired state?

You need to know the number of NICs, for example for three of them this would be it.

Thanks. Having a Python-like `0:-1` slice index to select a sublist could be even cooler.

More importantly, I would like to be able to select all interfaces which have vlan 204 enabled on their switch port, as reported by lldp. What would the capture look like in this case?

capture:
vlan-204: interfaces.lldp.enabled=="true" | interfaces.lldp.neighbors.ieee-802-1-vlans.vid==204

But I suspect our current implementation will fail resolving this capture since not all the neighbor entries have the "ieee-802-1-vlans" key.

We can change the implementation not to fail if at least one of the entries in a list has the required path.

Then, even the following capture should work-

capture:
vlan-204: interfaces.lldp.neighbors.ieee-802-1-vlans.vid==204

yes

Felix Enrique Llorente Pastora

unread,

Dec 20, 2021, 3:42:45 AM12/20/21

to Alona Paz, Dan Kenigsberg, kubevirt-dev, nmstat...@lists.fedorahosted.org

El dom., 19 dic. 2021 10:20 a. m., Alona Paz <alka...@redhat.com> escribió:

On Sun, Dec 19, 2021 at 10:42 AM Dan Kenigsberg <dan...@redhat.com> wrote:

On Fri, Dec 17, 2021 at 5:25 PM Felix Enrique Llorente Pastora <ello...@redhat.com> wrote:

On Fri, Dec 17, 2021 at 3:58 PM Dan Kenigsberg <dan...@redhat.com> wrote:
Thanks for sharing your plan.

I would like to create a Linux bond of all the NICs of my machine, create a Linux bridge on top of it, and copy my original mac/IP/routing to that bridge. Can you share a policy that would let me express this desired state?

You need to know the number of NICs, for example for three of them this would be it.

Thanks. Having a Python-like `0:-1` slice index to select a sublist could be even cooler.

Usually this is implemented with a "map" function that transform one list into another passing the current list item and doing some tranformation, in this case

we have to transform the interfaces items into ports, I have open an issue at nmpolicy for implement that we some special syntax, https://github.com/nmstate/nmpolicy/issues/72.

Dan Kenigsberg

unread,

Dec 28, 2021, 3:40:04 AM12/28/21

to Alona Paz, Felix Enrique Llorente Pastora, kubevirt-dev, nmstat...@lists.fedorahosted.org

On Sun, Dec 19, 2021 at 11:20 AM Alona Paz <alka...@redhat.com> wrote:
>
>
>
> On Sun, Dec 19, 2021 at 10:42 AM Dan Kenigsberg <dan...@redhat.com> wrote:
>>
>>
>>
>> On Fri, Dec 17, 2021 at 5:25 PM Felix Enrique Llorente Pastora <ello...@redhat.com> wrote:
>>>
>>>
>>>
>>> On Fri, Dec 17, 2021 at 3:58 PM Dan Kenigsberg <dan...@redhat.com> wrote:
>>>>
>>>> Thanks for sharing your plan.
>>>>
>>>> I would like to create a Linux bond of all the NICs of my machine, create a Linux bridge on top of it, and copy my original mac/IP/routing to that bridge. Can you share a policy that would let me express this desired state?
>>>
>>>
>>> You need to know the number of NICs, for example for three of them this would be it.
>>
>>
>> Thanks. Having a Python-like `0:-1` slice index to select a sublist could be even cooler.
>>
>> More importantly, I would like to be able to select all interfaces which have vlan 204 enabled on their switch port, as reported by lldp. What would the capture look like in this case?
>
>
> capture:
> vlan-204: interfaces.lldp.enabled=="true" | interfaces.lldp.neighbors.ieee-802-1-vlans.vid==204
>
> But I suspect our current implementation will fail resolving this capture since not all the neighbor entries have the "ieee-802-1-vlans" key.
> We can change the implementation not to fail if at least one of the entries in a list has the required path.
>
> Then, even the following capture should work-
>
> capture:
> vlan-204: interfaces.lldp.neighbors.ieee-802-1-vlans.vid==204

Thanks. I find the capture language very powerful, I think I would
benefit from a tool that helps me build them, based on the current
state.
E.g a command line such as
nmstatectl show | nmpolicy testcapture
'interfaces.lldp.enabled=="true" |
interfaces.lldp.neighbors.ieee-802-1-vlans.vid==204'
that would emit the value of yaml-encoded captures.

Would something like this be possible?

Felix Enrique Llorente Pastora

unread,

Jan 3, 2022, 1:42:59 AM1/3/22

to Dan Kenigsberg, Alona Paz, kubevirt-dev, nmstat...@lists.fedorahosted.org

You can inspect that information at ~/.cache/nmpolicy/cache.yaml, it will dump there the captured states.

--

Dan Kenigsberg

unread,

Jan 3, 2022, 2:38:31 AM1/3/22

to Felix Enrique Llorente Pastora, Alona Paz, kubevirt-dev, nmstat...@lists.fedorahosted.org

On Mon, Jan 3, 2022 at 8:42 AM Felix Enrique Llorente Pastora

I see a few drawbacks to relying on this:
- caches are typically not visible to end users, their content and
location may change
- I need to write a full policy; I prefer to focus on a single capture string
- When I run the nmpolicyctl I create a side effect (changing the
cache). I don't want to do so when I test captures. I'm looking for a
pure test-only usage.

P.s. because we keep the cache for a long while, we need to place a
version number in it, in case we ever need to make a breaking change
in its format.

Regards,
Dan.

Felix Enrique Llorente Pastora

unread,

Jan 3, 2022, 2:52:21 AM1/3/22

to Dan Kenigsberg, Alona Paz, kubevirt-dev, nmstat...@lists.fedorahosted.org

Right, maybe eve a nmpolicyctl filter make sense beyond testing purpose, it will allow the user to filter state, like stuff we do a knnstate filtering veth.

Reply all

Reply to author

Forward