Installed Submariner on Kind clusters (singlenode both with weavenet) failed

[Zayn Khan]

Hi

I have Installed submariner on two kind clusters with weavenet as CNI. When I deployed on broker on kind cluster 1 and join the cluster1 and cluster2 to broker. So there are we were no pods except operator pods. I have subctl gather file and some screenshots from subctl show all command.

Can somebody help on that?

[Aswin Suryanarayanan]

Hi,

Seems like the broker is not reachable even from cluster1.  Was the join run against an old broker-info.subm file by any chance?

Also did you try the diagnose command?

Thanks

Aswin

--
You received this message because you are subscribed to the Google Groups "submariner-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to submariner-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/submariner-users/f64b3656-81af-4432-b0cc-bc6050108ec1n%40googlegroups.com.

[Zayn Khan]

Hi Aswin,

Thanks for reaching out.

No, the broker-info.subm file is created newly on cluster a and then I ran the subctl join for the cluster a and cluster b. I have the attached the logs on this email thread can you please have a look this. 

This issue is persistent with all my testing environments like previously I have created the two K3s cluster and deploy the broker on first cluster and then do the same join procedure for both clusters but the issue was same there were only a single pod operator pod and no other pod was there. 

[Janki Chhatbar]

Hi Zayn

How are you installing Broker and Submariner? If you are using `subctl`, can you please ensure that the `subctl` version is the same as `Submariner` version you are trying to install.

Thanks

Janki

To view this discussion on the web visit https://groups.google.com/d/msgid/submariner-users/0403db0e-3c97-40e3-9644-07e47bdb4c0en%40googlegroups.com.

[Zayn Khan]

Hi Janki,

I am installing submariner through subctl. I have run the command subctl version which is ----------> v0.14.4. Any then I ran the command the subctl show all for the operator version which is also v0.14.4.
I am attaching screen shots of subctl command ----> $subctl diagnose all. Please have a look at it I have taken this screenshot after a fresh install of broker on cluster1 and then join the cluster1 and cluster2 to broker.

Many Thanks

Zayn

[Janki Chhatbar]

I had the exact same error but my subctl and submariner versions were different. Can you try re-running `join` on both clusters. Also Submariner 0.15.0 is now available. Any reason you are installing 0.14.4?

Thanks

Janki

To view this discussion on the web visit https://groups.google.com/d/msgid/submariner-users/208a4883-bf91-4409-8967-976a9311dbc7n%40googlegroups.com.

[Zayn Khan]

No I am not installing the 0.14.4. Now it is by default the 0.14.4 when I join the clusters. Can you please show me the correct steps how to properly match the versions of both submariner and subctl.

[Janki Chhatbar]

Thanks

Janki

On Mon, May 22, 2023 at 12:00 PM Zayn Khan <engr....@gmail.com> wrote:

No I am not installing the 0.14.4. Now it is by default the 0.14.4 when I join the clusters. Can you please show me the correct steps how to properly match the versions of both submariner and subctl.

That is because your subctl version is 0.14.4. YOu can install 0.15.0 subctl by running  `curl -Ls https://get.submariner.io | VERSION=v0.15.0 bash` and then re-run deploy and join. This will install 0.15.0v of Submariner.

To view this discussion on the web visit https://groups.google.com/d/msgid/submariner-users/cf955a42-1a48-4715-971d-f2d18a3c546fn%40googlegroups.com.

[Zayn Khan]

Janki, Let me try this. Back to you in a while.

[Zayn Khan]

I have installed the latest version of subctl ---> v0.15.0 and the deploy the broker and join process of clusters. But the issue is still persistent.
I have attached a screen shot of diagnose all. Please have a look at it

On Monday, May 22, 2023 at 11:36:44 AM UTC+5 jchh...@redhat.com wrote:

[Sridhar Gaddam]

Hello Zayn,

I can see that you are having two clusters (kind-cluster1 and kind-cluster2) each one with a single node.

It is not clear if the Broker cluster is one of them or if it's a totally different cluster. 

The reason the Submariner pods are not up and running is because the submariner-operator pod is unable to connect with the Broker cluster. 

I had a look at the logs, but they are redacted, so will not be able to tell the IPaddress of the Broker from the logs - so please check if your Broker cluster (ie.., the cluster against which you ran the subctl deploy-broker command) is accessible from the two KIND clusters - it looks like its not.

 [90m2023-05-19T17:14:06.239Z [0m  [1m [31mERR [0m [0m ..oller/controller.go:326                      Reconciler error  [36merror= [0m [31m"error building an authorized RestConfig for the broker: Get \"https://##redacted-api-server##/api/v1/namespaces/submariner-k8s-broker/secrets/any\": dial tcp ##redacted-api-server##: connect: connection refused" [0m  [36mcontroller= [0msubmariner-controller  [36mcontrollerGroup= [0msubmariner.io 

I'm not sure if you have any specific reason for installing weavenet, but Submariner has necessary scripts to deploy KIND clusters with kindnet CNI and the steps are documented on the website.

https://submariner.io/getting-started/quickstart/kind/

 

Best Regards,

--Sridhar.

To view this discussion on the web visit https://groups.google.com/d/msgid/submariner-users/21bc3e3d-aa75-48ff-9a7d-51f18a3db87bn%40googlegroups.com.

[Zayn Khan]

Hi Sridhar,

Yes I have followed the sandbox tutorial which is working fine on one of my VM. The main Idea behind was this I have two production k8s cluster with weavenet I dont want to take any risk on the production cluster In case any mishap occurs the services will goes down.

Thats why I have replicated the Sandbox environment scenario on the two kind clusters with cni weave and followed the steps of deployment of broker on Kind Cluster1 and the join the cluster1 and cluster 2 to it. After all this when I ran the subctl diagnose all it shows error list in above screen shot.
Can you please correct me what I am doing wrong.

Thanks

Zayn

[Sridhar Gaddam]

Couple of questions:

1. Is your Sandbox environment on the same Host?

2. Among the two KIND clusters, which one is used as Broker?

3. From the logs you shared, the problem is that the submariner-operator is not able to reach the Broker cluster either due to some firewall configuration or due to the way clusters are deployed.

4. If your Broker is on one of the KIND clusters, I would expect that at least there should not be an error in that KIND cluster to reach the Broker (as it's the same cluster). But I see that both the KIND clusters are not able to reach the Broker.

4. Since you mentioned that this is a Sandbox env', if you are okay to share, you can re-run the `subctl gather --include-sensitive-data ` which includes the IPAddress and other details from which we can tell the Broker IP that is not reachable.

It would be easier to troubleshoot over slack. You can reach out to the Submariner team at https://submariner.io/community/getting-help/#slack

Best Regards,

--Sridhar.

To view this discussion on the web visit https://groups.google.com/d/msgid/submariner-users/e2be7360-eeee-4d23-9988-f697fad8ea4dn%40googlegroups.com.

[Zayn Khan]

Hi Sridhar,

1. No I was using a different VM for my customized sandbox environment.

2. Cluster 1 is broker.

The issue has been resolved by :

Kind usually generate a kubeconfig file with loopback address 127.0.0.1:someport for clusters at local host on $HOME/.kube/config. One thing I was missing to change the server ip of both cluster to container ip on which clusters are running. Every thing was fine then.

I thankyou all the submariner team for helping me out on the issue. I have joined the slack channel next time I will post the issue there.

Many Thanks,

Zayn

[Sridhar Gaddam]

Hello Zayn,

Glad to hear that it's working now!!

Yes, when the KIND clusters are deployed using Submariner scripts, the server ip is updated here

https://github.com/submariner-io/shipyard/blob/13f4a54a035d05dc6428d61a0d241253590255c2/scripts/shared/lib/clusters_kind#L46

Best Regards,

--Sridhar.

To view this discussion on the web visit https://groups.google.com/d/msgid/submariner-users/e270b8f0-b979-4cce-8bf7-104a538a7ea6n%40googlegroups.com.

[Zayn Khan]

Thanks Again for your valuable input.

Many Thanks,

Zayn