iptables and PBR with ip-rule

[Girish Moodalbail]

Hello all,

I wanted to check with you all whether my conclusion below is correct on iptables and policy based routing using ip-rule.

This is the logical topology I have

+-----------------------------------------+
| logical router |
| Static Route |
| (10.8.51.79 -- nexthop -->172.16.4.2) |
| |
+--------+-----------------+--------------+
| |
| |
+---------+--------+ +---+--------------+
| ls0 | | ls1 |
| (192.168.4.0/24) | | (172.16.4.0/24) |
+--+------+------+-+ +-------------+----+
| | | |
| | | |
ovn-k8s-mp0 | | |
192.168.4.2/24 v | ovn-lcl-p0
pod0 | 172.16.4.2/24
v
pod1


and all the logical entities above are in one single node as shown below.

+---------------------------------------------------+
|Node1 (10.8.51.79) |
| +-----+ +-----+ |
| |POD0 | |POD1 | |
| ovn-k8s-mp0 | | | | |
| 192.168.4.2/24 +-+---+ +--+--+ |
| | | | |
| | | | |
| | | | |
| +----+-----------+---------+--+ |
| ovn-lcl-p0 | br-int | |
|172.16.4.2/24------| (OVN Logical pipeline) | |
| | | |
| +-----------------------------+ |
+---------------------------------------------------+


I am trying to access a load balancer VIP of 10.96.0.200:30173 that is backed by the node's IP of 10.8.51.79:30173. I have defined the corresponding OVN LB rule ("10.96.0.200:30173"="10.8.51.79:30173"), and I have a route to 10.96.0.0/12 via dev ovn-k8s-mp0 on the node.

I have following iptables mangle rules on the node, which basically marks all the packets leaving `ovn-lcl-p0` towards the host from OVN pipeline with 0xfff and then restores the mark on the reply-packets on that connection

iptables -t mangle -A PREROUTING -s 192.168.0.0/16 -i ovn-lcl-p0 -m mark --mark 0x0 -j MARK --set-xmark 0xfff
iptables -t mangle -A PREROUTING -j CONNMARK --save-mark
iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark

Next, I have the ip-rule below which routes such marked packets through the interface it came on (ovn-lcl-po). If I don't have the rule, the packets will go through ovn-k8s-mp0 since it has more direct route for 192.168.4.0/24.

ip rule add fwmark 0xfff table 42
ip route add 192.168.0.0/16 via 172.16.4.1 table 42


What works?
~~~~~~~~~~~
If I try to connect to the LB VIP from within the pod, everything works. The packet starts off with
1. (src:192.168.4.3, dst:10.96.0.200)
2. gets dnat'ed in LS0 --> (src:192.168.4.3, dst:10.8.51.79)
3. hits the logical_router_static_route and gets forwarded to ovn-lcl-p0 --> (src:192.168.4.3, dst:10.8.51.79)
4. hits the iptables mangle rules, gets marked, reaches the process on the host and packets are steered back into the OVN pipeline through table 42

What doesn't work?
~~~~~~~~~~~~~~~~~~~
If I try to connect to the LB VIP from the host, it doesn't work
1. (src:192.168.4.2, dst:10.96.0.200)
2. gets dnat'ed in LS0 --> (src:192.168.4.2, dst:10.8.51.79)
3. hits the logical_router_static_route and gets forwarded to ovn-lcl-p0 --> (src:192.168.4.2, dst:10.8.51.79)
4. hits the iptables mangle rules and gets marked. I only see SYN packets but no SYN-ACK

My thinking is that, both the source and destination IP address are defined on the host stack. So, routes for all the directly attached interface are in `local` route table, that has higher priority than the table 42, and will never hit table 42.

# ip rule show
0: from all lookup local
1000: from fwmark 0xfff lookup 42
32766: from all lookup main
32767: from all lookup default

The only way to get this to work is to SNAT the packets coming out of ovn-lcl-p0?

Regards,
~Girish





-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------