overlay IP leaks to the underlay

[Yun Zhou]

Hi,

In our ovn-k8s setup, we've been seeing that occasionally overlay IP was leaked to the underlay interface. Specifically, a pod (whose overlay IP is 10.192.8.24) was trying to connect to an external IP 176.32.112.172, In theory, the outgoing packet should be SNATed to the underlying gateway interface IP (10.0.1.13), but for some reason, one of its FIN packet was sent out with source IP 10.192.8.24. See attached tcpdump.ens4f0.txt.

Some more information to add from our observation:

1. The problem is not persistent all the time, it is only shown occasionally.
2. the problematic packet is always the 1st FIN packet sent (the 2nd FIN does not have any problem).
3. ovs-vswitchd.log seems to suggest this FIN packet was missing the upcall: (below snip is from a different run):

"2021-03-23T16:43:45.087Z|9973291|dpif(handler1)|DBG|system@ovs-system: miss upcall:
recirc_id(0x5820c1),dp_hash(0),skb_priority(0),in_port(12),skb_mark(0),ct_state(0x30),ct_zone(0x90),ct_mark(0),ct_label(0),eth(src=0a:58:64:40:00:01,dst=0a:58:64:40:00:1d),eth_type(0x0800),ipv4(src=10.192.8.24,dst=176.32.112.172,proto=6,tos=0,ttl=63,frag=no),tcp(src=55034,dst=443),tcp_flags(fin|ack)
tcp,vlan_tci=0x0000,dl_src=0a:58:64:40:00:01,dl_dst=0a:58:64:40:00:1d,nw_src=10.192.8.24,nw_dst=176.32.112.172,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=55034,tp_dst=443,tcp_flags=fin|ack tcp_csum:9b3"

What could be the reason why this happens? If the provided information is not enough, please let me know what else is needed and I will try to collect.

I've attached something we've been collecting:

1. tcpdump.ens4f0.txt: tcpdump output of the problematic connection
2. conntrack_event.txt: conntrack event
3. dpctl_dump_flow.txt: ovs-dpctl dump-flows output (got after seeing the issue, of a different run)

Thanks
- Cathy