L2 Mode from within a subnet

[ghorofa M]

I have a functional Metallb  installation on a ks8 cluster, with public ips working from the public interface as expected.

On the other hand i have some 172.18.6.xx addresses given to internal services that would be useful to access over the L3 network.  Ks8 is installed bound to  interface  wg0,

with nodes in 172.18.6.10-20,    and some external IPs given to services on  .150-200, 

the master is connected to the LAN, which is on .220-230. The external ips are accessible from within the cluster: 

curl 172.18.6.150   works

however,  from  the rest of the lan,  say workstation 172.18.6.220, which is connected to the master  via interface wg0,  no response, however the traffic reaches the master, and is visible on tcpdump on the master:

14:33:28.133681 wg0   In  IP 172.18.6.120.32984 > 172.18.6.150.http: Flags [S], seq 406428854, win 64860, options [mss 1380,sackOK,TS val 1873125555 ecr 0,nop,wscale 7], length 0
14:33:29.237091 wg0   In  IP 172.18.6.150.42564 > 172.18.6.150.http: Flags [S], seq 90733823, win 64860, options [mss 1380,sackOK,TS val 1873126658 ecr 0,nop,wscale 7], length 0

Conntrack shows no reply sent

# conntrack -E | grep 172.18.6.150
    [NEW] tcp      6 120 SYN_SENT src=172.18.6.122 dst=172.18.6.150 sport=48180 dport=80 [UNREPLIED] src=172.18.6.150 dst=PUBLIC_IP  sport=80 dport=48180

How would i get a reply here? This is L2 mode.  the interface wg0 is L3 (wireguard), does this pose a problem?

I saw a note to the effect "layer 2 mode relies on ARP and NDP, the client must be on the same subnet of the nodes announcing the service in order for MetalLB to work",

does this mean outside the cluster i have no luck with the L2 addresses over wireguard?

[Elias Morais Pereira]

Hello ghorofa,

How did you configure the public IP on metalllb? Could you can share your configuration?

[Pankaj Agarwal]

It’s not the client but the router in front of the node that has this IP you are referring to as Public here.

The metallb ARP announcement is good for the router in front to know that traffic for this IP needs to be routed to this node but in your specific case other routers might not be aware of the same.

You can do this, have your network team route all traffic this IP or IP block to a certain router and then limit matallb to the nodes behind this router.

I am assuming you do not have BGP routing as an option at your network as that might be a better/easier option too.

I hope this helps.

Thanks,

Pankaj

Sent from my iPhone

On Dec 10, 2022, at 3:50 PM, 'Elias Morais Pereira' via metallb-users <metall...@googlegroups.com> wrote:

Hello ghorofa,

--
You received this message because you are subscribed to the Google Groups "metallb-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to metallb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/metallb-users/cacaabbc-1736-4d9e-bed2-e1382a2aa648n%40googlegroups.com.

[Elias Morais Pereira]

hello Pankaj, thanks for the answer!!!

I thought I could set up a public IP pool directly on metallb, but I believe that to
work this way, the k8s cluster should also be on this network.

As I thought of a better security for the cluster, I left it in a specific vlan with
private IP. I could also expose the cluster via public IP, but would be many IPs allocated.

I think I misunderstood the way metallb works with the network. I thought it would 

be easier to configure it only with public IP. The big problem is to depend on NAT,

routing all the private network to go out to the Internet with public IP.

In the cloud I know that metallb is not necessary, because each broker has its load-balance

system and allocates an IP or a pool of public IPs to the LB. In the cloud the nodes of the

cluster also receive a public IP or can stay with a private IP?