Submariner Gateways Not Establishing Connections Between Clusters

[hala kammoun]

• I'm experiencing an issue where the Submariner gateways between my clusters are not establishing connections as expected and gets stuck in the Connecting status (the ipsec tunnel is not established) . 
• When running subctl show gateways, I observed that the gateways in member1 and member2 clusters are active, but 0 connections out of 1 are established for each.

Below is a summary of the scenario and the troubleshooting steps I have taken so far.

• i'm using 3 clusters with minikube profile (each cluster have isolated network) for karmada and 2 members, setting also different service and pod CIDRs to avoid overlapping .

this is the subctl gather output

hale@hale-VirtualBox:~$ subctl gather

Cluster "karmada-cluster"

Gathering information from cluster "karmada-cluster"

 ✓ Gathering broker logs

 ✓ Gathering broker resources

 ⚠ Gathering operator logs

 ✓ Found 1 pods matching label selector "name=submariner-operator"

 ⚠ Found logs for previous instances of pod submariner-operator-587b7f4fc6-frhq2

 ✓ Gathering operator resources

 ✓ Found 0 submariners in namespace "submariner-operator"

 ✓ Found 0 servicediscoveries in namespace "submariner-operator"

 ✓ Found 1 deployments by field selector "metadata.name=submariner-operator" in namespace "submariner-operator"

 ✓ Found 0 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"

 ✓ Found 0 daemonsets by label selector "app=submariner-metrics-proxy" in namespace "submariner-operator"

 ✓ Found 0 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"

 ✓ Found 0 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"

 ✓ Found 0 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"

 ✓ Found 0 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"

 ⚠ Gathering connectivity logs

 ⚠ The Submariner connectivity components are not installed

 ⚠ Gathering connectivity resources

 ⚠ The Submariner connectivity components are not installed

 ⚠ Gathering service-discovery logs

 ⚠ The Submariner service discovery components are not installed

 ⚠ Gathering service-discovery resources

 ⚠ The Submariner service discovery components are not installed

Files are stored under directory "submariner-20240907184724/karmada-cluster"



Cluster "member1"

Gathering information from cluster "member1"

 ✓ Gathering broker logs

 ✓ Gathering broker resources

 ✓ Found 2 endpoints in namespace "submariner-k8s-broker"

 ✓ Found 2 clusters in namespace "submariner-k8s-broker"

 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace "submariner-k8s-broker"

 ✓ Found 0 serviceimports in namespace "submariner-k8s-broker"

 ✓ Gathering operator logs

 ✓ Found 1 pods matching label selector "name=submariner-operator"

 ✓ Gathering operator resources

 ✓ Found 1 submariners in namespace "submariner-operator"

 ✓ Found 1 servicediscoveries in namespace "submariner-operator"

 ✓ Found 1 deployments by field selector "metadata.name=submariner-operator" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-metrics-proxy" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"

 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"

 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"

 ✓ Gathering connectivity logs

 ✓ Found 1 pods matching label selector "app=submariner-gateway"

 ✓ Found 1 pods matching label selector "app=submariner-routeagent"

 ✓ Found 1 pods matching label selector "app=submariner-metrics-proxy"

 ✓ Found 1 pods matching label selector "app=submariner-metrics-proxy"

 ✓ Found 1 pods matching label selector "app=submariner-globalnet"

 ✓ Found 0 pods matching label selector "app=submariner-addon"

 ✓ Gathering connectivity resources

 ✓ Gathering CNI data from 1 pods matching label selector "app=submariner-routeagent"

 ✓ Gathering CNI data from 1 pods matching label selector "app=submariner-gateway"

 ✓ Gathering cable driver data from 1 pods matching label selector "app=submariner-gateway"

 ✓ Found 2 endpoints in namespace "submariner-operator"

 ✓ Found 2 clusters in namespace "submariner-operator"

 ✓ Found 1 gateways in namespace "submariner-operator"

 ✓ Found 1 clusterglobalegressips in namespace ""

 ✓ Found 0 globalegressips in namespace ""

 ✓ Found 0 globalingressips in namespace ""

 ⚠ Gathering service-discovery logs

 ✓ Found 3 pods matching label selector "component=submariner-lighthouse"

 ✓ Found 1 pods matching label selector "k8s-app=kube-dns"

 ⚠ Found logs for previous instances of pod coredns-7db6d8ff4d-9cl6p

 ✓ Gathering service-discovery resources

 ✓ Found 0 serviceexports in namespace ""

 ✓ Found 0 serviceimports in namespace ""

 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace ""

 ✓ Found 1 configmaps by label selector "component=submariner-lighthouse" in namespace "submariner-operator"

 ✓ Found 1 configmaps by field selector "metadata.name=coredns" in namespace "kube-system"

 ✓ Found 0 services by label selector "submariner.io/exportedServiceRef" in namespace ""

Files are stored under directory "submariner-20240907184724/member1"



Cluster "member2"

Gathering information from cluster "member2"

 ✓ Gathering broker logs

 ✓ Gathering broker resources

 ✓ Found 2 endpoints in namespace "submariner-k8s-broker"

 ✓ Found 2 clusters in namespace "submariner-k8s-broker"

 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace "submariner-k8s-broker"

 ✓ Found 0 serviceimports in namespace "submariner-k8s-broker"

 ✓ Gathering operator logs

 ✓ Found 1 pods matching label selector "name=submariner-operator"

 ✓ Gathering operator resources

 ✓ Found 1 submariners in namespace "submariner-operator"

 ✓ Found 1 servicediscoveries in namespace "submariner-operator"

 ✓ Found 1 deployments by field selector "metadata.name=submariner-operator" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-metrics-proxy" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"

 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"

 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"

 ✓ Gathering connectivity logs

 ✓ Found 1 pods matching label selector "app=submariner-gateway"

 ✓ Found 1 pods matching label selector "app=submariner-routeagent"

 ✓ Found 1 pods matching label selector "app=submariner-metrics-proxy"

 ✓ Found 1 pods matching label selector "app=submariner-metrics-proxy"

 ✓ Found 1 pods matching label selector "app=submariner-globalnet"

 ✓ Found 0 pods matching label selector "app=submariner-addon"

 ✓ Gathering connectivity resources

 ✓ Gathering CNI data from 1 pods matching label selector "app=submariner-routeagent"

 ✓ Gathering CNI data from 1 pods matching label selector "app=submariner-gateway"

 ✓ Gathering cable driver data from 1 pods matching label selector "app=submariner-gateway"

 ✓ Found 2 endpoints in namespace "submariner-operator"

 ✓ Found 2 clusters in namespace "submariner-operator"

 ✓ Found 1 gateways in namespace "submariner-operator"

 ✓ Found 1 clusterglobalegressips in namespace ""

 ✓ Found 0 globalegressips in namespace ""

 ✓ Found 0 globalingressips in namespace ""

 ⚠ Gathering service-discovery logs

 ✓ Found 3 pods matching label selector "component=submariner-lighthouse"

 ✓ Found 1 pods matching label selector "k8s-app=kube-dns"

 ⚠ Found logs for previous instances of pod coredns-7db6d8ff4d-p262h

 ✓ Gathering service-discovery resources

 ✓ Found 0 serviceexports in namespace ""

 ✓ Found 0 serviceimports in namespace ""

 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace ""

 ✓ Found 1 configmaps by label selector "component=submariner-lighthouse" in namespace "submariner-operator"

 ✓ Found 1 configmaps by field selector "metadata.name=coredns" in namespace "kube-system"

 ✓ Found 0 services by label selector "submariner.io/exportedServiceRef" in namespace ""

Files are stored under directory "submariner-20240907184724/member2"
this is the subctl diagnose all output

hale@hale-VirtualBox:~$ subctl diagnose all

Cluster "karmada-cluster"

 ✓ Checking Submariner support for the Kubernetes version

 ✓ Kubernetes version "v1.30.0" is supported





 ⚠ Submariner connectivity feature is not installed



 ⚠ Submariner service discovery feature is not installed





Cluster "member1"

 ✓ Checking Submariner support for the Kubernetes version

 ✓ Kubernetes version "v1.30.0" is supported



 ✓ Globalnet deployment detected - checking that globalnet CIDRs do not overlap

 ✓ Checking DaemonSet "submariner-gateway"

 ✓ Checking DaemonSet "submariner-routeagent"

 ✓ Checking DaemonSet "submariner-globalnet"

 ✓ Checking DaemonSet "submariner-metrics-proxy"

 ✓ Checking Deployment "submariner-lighthouse-agent"

 ✓ Checking Deployment "submariner-lighthouse-coredns"

 ✓ Checking the status of all Submariner pods

 ✓ Checking that gateway metrics are accessible from non-gateway nodes

 ✓ Skipping this check as it's a single node cluster

 ✓ Checking that globalnet metrics are accessible from non-gateway nodes

 ✓ Skipping this check as it's a single node cluster



 ⚠ Checking Submariner support for the CNI network plugin

 ⚠ Submariner could not detect the CNI network plugin and is using ("generic") plugin. It may or may not work.

 ✗ Checking gateway connections

 ✗ Connection to cluster "member2" is in progress

 ✓ Checking Submariner support for the kube-proxy mode

 ✓ The kube-proxy mode is supported

 ✓ Checking that firewall configuration allows intra-cluster VXLAN traffic

 ✓ Skipping this check as it's a single node cluster

 ✓ Checking that Globalnet is correctly configured and functioning



 ✓ Checking that services have been exported properly



Cluster "member2"

 ✓ Checking Submariner support for the Kubernetes version

 ✓ Kubernetes version "v1.30.0" is supported



 ✓ Globalnet deployment detected - checking that globalnet CIDRs do not overlap

 ✓ Checking DaemonSet "submariner-gateway"

 ✓ Checking DaemonSet "submariner-routeagent"

 ✓ Checking DaemonSet "submariner-globalnet"

 ✓ Checking DaemonSet "submariner-metrics-proxy"

 ✓ Checking Deployment "submariner-lighthouse-agent"

 ✓ Checking Deployment "submariner-lighthouse-coredns"

 ✓ Checking the status of all Submariner pods

 ✓ Checking that gateway metrics are accessible from non-gateway nodes

 ✓ Skipping this check as it's a single node cluster

 ✓ Checking that globalnet metrics are accessible from non-gateway nodes

 ✓ Skipping this check as it's a single node cluster



 ⚠ Checking Submariner support for the CNI network plugin

 ⚠ Submariner could not detect the CNI network plugin and is using ("generic") plugin. It may or may not work.

 ✗ Checking gateway connections

 ✗ Connection to cluster "member1" is in progress

 ✓ Checking Submariner support for the kube-proxy mode

 ✓ The kube-proxy mode is supported

 ✓ Checking that firewall configuration allows intra-cluster VXLAN traffic

 ✓ Skipping this check as it's a single node cluster

 ✓ Checking that Globalnet is correctly configured and functioning



 ✓ Checking that services have been exported properly



Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.



subctl version: v0.18.0


and this is how i deploy the broker then how i join members

# deploy the broker in the karmada cluster

sudo /home/hale/.local/bin/subctl deploy-broker --globalnet --kubeconfig .kube/config --context karmada-cluster



# join member1 to the broker

sudo /home/hale/.local/bin/subctl join --kubeconfig .kube/config --context member1 broker-info.subm --natt=false --health-check=false



# join member2 to the broker

sudo /home/hale/.local/bin/subctl join --kubeconfig .kube/config --context member2 broker-info.subm --natt=false --health-check=false

I’m looking for recommendations or assistance on how to set up locally these clusters differently to avoid this gateway creation issue.

Thank you for your support!

[Nir Yechiel]

Hi Hala,

Thanks for trying Submariner. Is it possible for you to raise this issue on the #submariner Slack channel? I think you would get a faster reply, and perhaps we can help you troubleshoot the issue in a bit more interactive way.

Thanks,

Nir

--
You received this message because you are subscribed to the Google Groups "submariner-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to submariner-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/submariner-users/848587cf-65ae-464f-8eeb-0c2c1d34cfa5n%40googlegroups.com.

[Nir Yechiel]

Sorry, hit send too quickly. I also wanted to suggest that you report an Issue on GitHub for proper tracking.

Thanks

Nir

[hala kammoun]

Thanks, I'll look into it.