lb_force_snat_ip and NetworkPolicy

[Girish Moodalbail]

Hello all,

The lb_force_snat_ip on the GR changes the source address of the packet arriving from the external network to the GR's internal IP in the 100.64.X.Y/29 subnet. So, when the packet arrives at the destination pod on the overlay network, it would have lost the original source IP address.

Now, how would one define the Ingress Network Policy for that Pod? Say, these are the configurations:

K8s Node Subnets: 10.8.48.0/24
GR to DR Join Switch Subnets: 100.64.0.0/16 (divided into /29 chunks)
Cluster Subnet: 192.168.0.0/16

Consider the workload below.

Cluster Subnet
192.168.0.0/16
+------------------------------+
|Web Deployment (replicas 3) |
|+--------++--------++--------+|
|| web || web || web ||
|| pod1 || pod2 || pod3 ||
|| 4.5 || 5.6 || 7.9 ||
|+--------++--------++--------+|
+--------------^---------------+
|
|
|
+--------------+---------------+
| Kubernetes Service of |
| Type LoadBalancer |
| (and NodePort Service) |
+------------------------------+


So, if a NetworkPolicy for the web pods above is defined with the ingress as below...

ingress:
- from:
- ipBlock:
cidr: 10.8.48.0/24
ports:
- protocol: TCP
port: 80

...it is not of any use, since when the packet arrives at one of the POD, the src IP will be from 100.64.0.0/16 subnet. If we fix the above ingress by using the ingress as below...

ingress:
- from:
- ipBlock:
cidr: 100.64.0.0/16
ports:
- protocol: TCP
port: 80

... it will work, but then again the tenants will now need to know about 100.64.0.0/16 subnet which was supposed to be abstracted for them.

Also, say you want to restrict the pod access from only certain section of the DC (some subnet say 172.16.1.0/24), then we will not be able to do so since all the packets gets SNATed to 100.64.0.0/16?

Am I missing something here?

Regards,
~Girish

[Dan Winship]

On 5/21/20 11:28 PM, Girish Moodalbail wrote:
> Hello all,
>
> The lb_force_snat_ip on the GR changes the source address of the packet arriving from the external network to the GR's internal IP in the 100.64.X.Y/29 subnet. So, when the packet arrives at the destination pod on the overlay network, it would have lost the original source IP address.

Note that the Kubernetes docs explicitly warn that this might happen:

Cluster ingress and egress mechanisms often require rewriting
the source or destination IP of packets. In cases where this
happens, it is not defined whether this happens before or after
NetworkPolicy processing, and the behavior may be different for
different combinations of network plugin, cloud provider,
Service implementation, etc.

In the case of ingress, this means that in some cases you may be
able to filter incoming packets based on the actual original
source IP, while in other cases, the “source IP” that the
NetworkPolicy acts on may be the IP of a LoadBalancer or of the
Pod’s node, etc.

(From
https://kubernetes.io/docs/concepts/services-networking/network-policies/#behavior-of-to-and-from-selectors)

So while we could definitely be doing better than we are now, the
current behavior is not actually wrong.

-- Dan

PS - though if we could get rid of lb_force_snat_ip somehow that would
be great, since OVN currently only lets you define it for a single IP
family so we won't be able to make it work currently in dual-stack
clusters until that's fixed...

[Girish Moodalbail]

Thanks Dan for the explaination and pointer to the documentation.

BTW WRT to your PS, unfortunately I don't think we can get rid of lb_force_snat_ip since it is essential for achieving symmetrical return path so that the replies can be DNATed and send it back to the initiator. The PBR we have today is `stateless`, so we can't use that either.

On 5/22/20, 10:02 AM, "Dan Winship" <dwin...@redhat.com> wrote:

External email: Use caution opening links or attachments

[Tim Rozet]

Hey Girish,

Can't we apply ACL to source ip match before the ingress NAT happens?

Tim Rozet

Red Hat CTO Networking Team

--
You received this message because you are subscribed to the Google Groups "ovn-kubernetes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ovn-kubernete...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ovn-kubernetes/956F9538-27FD-425E-BF86-A9D4177DAB64%40nvidia.com.