[OVN][Scale] Conjunctive matches exponentially increase in Table 45

[Girish Moodalbail]

Hello all,

 

If a K8s NetworkPolicy is defined as below...

 

kind: NetworkPolicy

metadata:

  name: allow_from_within_namespace

  namespace: ns1

spec:

  podSelector: {}

  ingress:

  - from:

    - podSelector: {}

 

... which basically allows all packets from all Pods to all the Pods within the same namespace (ns1) then ovn-k8s translates that to the following OVN ACL...

 

"ip4.src == {$address_set_name} && outport == @port_group_name"

 

...which basically says all packets with source ip from logical_switch_ports in ns1 to logical_switch_ports in ns1 are allowed.

 

This particular OVN ACL results in an explosion of OpenFlow rules in Table 45 on each of the OVN Chassis in certain cases.

 

In the following case, no conjunctive match is used and there is no explosion.

 

1. Consider a K8s cluster with 10 nodes

2. There are 10 LSes (LS1, LS2, ... LS10), one per each node

3. Each node has one pod (aka LSP -- LSP1 on LS1, LSP2 on LS2, ....) in ns1 namespace

4. Table 45 doesn't contain any conjunction rules and everything is fine.

 

Extending the example above in the manner shown below results in addition of conjunction flows in Table 45.

 

5. Now schedule a 11th Pod on node1

6. That is, LSP11 on LS1

7. This results in addition of OpenFlow rule with conj_id field and two requirements for LS1

 

So, the conjunction is two-way
-----------8<--------------8<------

priority=2001,conj_id=4979,ip,reg0=0x80/0x80,metadata=LS1 actions=load:0x1->NXM_NX_XXREG0[97],resubmit(,46)

priority=2001,ip,reg0=0x80/0x80,metadata=LS1,nw_src=10.244.202.4 actions=conjunction(4979,1/2)

priority=2001,ip,reg0=0x80/0x80,metadata=LS1,nw_src=10.244.111.4 actions=conjunction(4979,1/2)

priority=2001,ip,reg0=0x80/0x80,metadata=LS1,nw_src=10.244.180.4 actions=conjunction(4979,1/2)

<7 more lines snipped>

-----------8<--------------8<------

Extending the example above in the manner shown below results in addition of more conjunction flows in Table 45.

 

8. Now schedule a 12th pod on node2

9. That is, LSP21 on LS2

10. This results in addition of NEW OpenFlow rule with conj_id field and two requirements for the LS2

 

So, the conjunction is two-way
-----------8<--------------8<------

priority=2001,conj_id=4980,ip,reg0=0x80/0x80,metadata=LS2 actions=load:0x1->NXM_NX_XXREG0[97],resubmit(,46)

priority=2001,ip,reg0=0x80/0x80,metadata=LS2,nw_src=10.244.202.4 actions=conjunction(4979,1/2)

priority=2001,ip,reg0=0x80/0x80,metadata=LS2,nw_src=10.244.111.4 actions=conjunction(4979,1/2)

priority=2001,ip,reg0=0x80/0x80,metadata=LS2,nw_src=10.244.180.4 actions=conjunction(4979,1/2)

<7 more lines snipped>

-----------8<--------------8<------ 

 

As you can see, addition of the 2nd port to each of the LS results in an exponential number of OpenFlow flows in Table 45.

 

Should the conjunction be with 3 requirements -- metadata, nw_src, and reg0? Right now, the conjunction is between nw_src and reg0.

 

Thanks,

~Girish

[Girish Moodalbail]

Essentially the number of OpenFlow rules in Table 45 can be calculated with the equation below:

 

OpenFlow rules := (Number of LS with more than one port * Number of conjunction requirements * number of LSP in port group)

 

If the cluster size is 1000 and that each of the LS has more than one port in the same namespace, then the number of OpenFlow rules would be:

 

1000 * 2 * 2000 = 4M flows

The thinking is if we make the conjunctive match requirements to be 3-way between -- metadata, reg0, and nw_src -- then the number of OpenFlow rules would be more linear.

Regards,

~Girish