3 Cluster setup Issue with Rancher

[çağatay erem]

Hello Team, 
I'm having an issue with 3 cluster setup with Rancher RKE1. Setup is consist of cluster A, B and C. A cluster has  3 nodes. B cluster is 1 node and C cluster is also 1 Node. Let me Clarify My setup and move on my issue.

With this Rancher setup, I m having issues with intra cluster communication with Cluster for A ( Since B is broker and C is single node, there are no intra cluster communication issues). There is no Firewall Limitation on my Cloud Provider. ( VMware based host provider). 

I wanna share all my rancher setup files, Log and every information as much as I can gather. 

I guess its worth to mention that Inter Cluster communication is actually working properly. I can Expose a service from Cluster A and it can be accessed from Cluster C but if its only in gateway node.So If  I expose a service which is not deployed on gateway node, I'm having this issue. 

I guess Its also worth to mention that I have tried expose a service on cluster A(not deployed on gateway Node), and tried to access from Cluster C. It does look like its Dns over subnet can be detected. But the issue looks like accessing into it. 

Here below,

In cluster C pod,

root@ubuntu-pod:/# wget nginx-service.default.svc.clusterset.local

--2023-12-14 15:18:14--  http://nginx-service.default.svc.clusterset.local/
Resolving nginx-service.default.svc.clusterset.local (nginx-service.default.svc.clusterset.local)... 10.45.77.107
Connecting to nginx-service.default.svc.clusterset.local (nginx-service.default.svc.clusterset.local)|10.45.77.107|:80... failed: No route to host.

In cluster A pod,

root@d4kcp-node02:/opt# kubectl get svc -o wide
NAME            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE     SELECTOR
kubernetes      ClusterIP   10.45.0.1      <none>        443/TCP   4h52m   <none>
nginx-service   ClusterIP   10.45.77.107   <none>        80/TCP    63m     app=nginx

So it seems that It does detect this cidr correctly. 

I have tried this setup with canal CNI, Flannel CNI and Calico CNI seperately, The result was the same. I have also changed the cluster names. It was local.cluster for 3 of my clusters. I have changed their names as one.cluster. two.cluster and so on. But the result still the same. 

I'm sharing the subctl gather output. I'm not sure why A cluster cannot communicate each other via Submariner. Route agents also doesn't tell much info. There are no issues shown on router or gateway logs. If It looks like a Bug, I can also move it into git as issue. 

I have kindly expect your helps. Thanks.