submariner in azure AKS

[Monish R]

Hi all,

we are trying out submariner in azure aks environment in two clusters, after following the helm installation steps provided in link 

https://submariner.io/operations/deployment/helm/ 

Note: our Env has,

CNI : kubenet / calico . 

We have not yet executed the post installation steps specific to calico CNI. assume that gateway should be up regardless.

we observe the deployed resources are not properly up in both cluster.

submariner-operator          pod/submariner-gateway-gwsv9                                        0/1     CrashLoopBackOff             38         170m

submariner-operator          deployment.apps/submariner-lighthouse-agent                       0/1     0            0           170m

submariner-operator          deployment.apps/submariner-lighthouse-coredns                     0/2     0            0           170m

Checking the logs on the submariner gateway :

I0810 17:56:36.097872       1 main.go:71] Starting the submariner gateway engine

W0810 17:56:36.098404       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.

I0810 17:56:36.098649       1 main.go:99] Creating the cable engine

I0810 17:56:36.311212       1 syncer.go:51] CableEngine syncer started

I0810 17:56:36.311821       1 main.go:263] Gateway leader election config values: main.leaderConfig{LeaseDuration:10, RenewDeadline:5, RetryPeriod:2}

I0810 17:56:36.311991       1 main.go:280] Using namespace "submariner-operator" for the leader election lock

I0810 17:56:36.312011       1 leaderelection.go:242] attempting to acquire leader lease  submariner-operator/submariner-engine-lock...

I0810 17:56:36.345107       1 leaderelection.go:252] successfully acquired lease submariner-operator/submariner-engine-lock

I0810 17:56:36.345144       1 main.go:133] Creating the tunnel controller

I0810 17:56:36.345143       1 event.go:281] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"submariner-operator", Name:"submariner-engine-lock", UID:"5730b78c-c596-43d2-b1a5-6212f62ec983", APIVersion:"v1", ResourceVersion:"1074527", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' aks-devfecpool-17806234-vmss000000-submariner-engine became leader

I0810 17:56:36.345184       1 main.go:141] Creating the kubernetes central datastore

I0810 17:56:36.345211       1 kubernetes.go:60] Rendered API server host: "https://13.XX.XX.XXX:443"

F0810 17:56:36.388633       1 main.go:313] Error creating the kubernetes datastore: error building submariner clientset: Get "https://13.XX.XX.XXX:443/apis/submariner.io/v1/namespaces/submariner-k8s-broker/endpoints": x509: certificate is valid for 10.XX.X.X, not 13.XX.XX.XXX

Any pointers in resolving this error is appreciated.

Regards,

R.Monish

[Vishal Thapar]

Hi Monish,

This looks like the API server endpoint given is 13.x.x.x while certificate is for 10.x.x.x  Can you can get API endpoint to use from your KUBECONFIG and try using that instead?

Regards,

Vishal.

--
You received this message because you are subscribed to the Google Groups "submariner-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to submariner-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/submariner-users/c168ca8c-085b-4e44-946e-78f04e63cc13n%40googlegroups.com.

[Monish R]

Hi Visha Thapar,

Thanks for the response.

As you have pointed out 13.x.x.x IP is the API endpoint from KUBECONFIG and  10.x.x.x  mentioned in the log is the service CIDR IP.

In which configuration do we need to make a change ?

Also when we ran : 

$ subctl diagnose all

✓ Checking Submariner support for the Kubernetes version used in cluster "cluster-B"

 ✓ The Kubernetes version meets Submariner's requirements

 ✓ Retrieving Submariner resource from " cluster-B "

 ✗ Checking Submariner support for the CNI network plugin in cluster " cluster-B "
 ✗ The detected CNI network plugin ("") is not supported by Submariner. Supported network plugins: [generic canal-flannel weave-net OpenShiftSDN OVNKubernetes]

The troubleshooting command does not detect AKS default CNI - kubenet/calico and the supported CNI in the list doesnt mention calico .

Is it fair to say submariner with calico is not yet supported ?

Regards,

R.Monish

[Vishal Thapar]

Hi Monish,

You will have to redeploy and modify this env variable [taken from helm install guide]

export SUBMARINER_BROKER_URL=$(kubectl -n default get endpoints kubernetes \ -o jsonpath="{.subsets[0].addresses[0].ip}:{.subsets[0].ports[?(@.name=='https')].port}")

IIRC, when you get it from endpoints it will return the 10.x.x.x IP. We should probably change this to pick IP from kubeconfig.

Let me know if this helps. You can reach out to us in our slack https://submariner.io/community/getting-help/#slack 

Regards,

Vishal.

[Sridhar Gaddam]

Hello Monish,

PSB inline.

On Thu, Aug 12, 2021 at 3:30 AM Monish R <monishop...@gmail.com> wrote:

Hi Visha Thapar,

Thanks for the response.

As you have pointed out 13.x.x.x IP is the API endpoint from KUBECONFIG and  10.x.x.x  mentioned in the log is the service CIDR IP.

In which configuration do we need to make a change ?

Also when we ran : 

$ subctl diagnose all

✓ Checking Submariner support for the Kubernetes version used in cluster "cluster-B"

 ✓ The Kubernetes version meets Submariner's requirements

 ✓ Retrieving Submariner resource from " cluster-B "

 ✗ Checking Submariner support for the CNI network plugin in cluster " cluster-B "
 ✗ The detected CNI network plugin ("") is not supported by Submariner. Supported network plugins: [generic canal-flannel weave-net OpenShiftSDN OVNKubernetes]

If you are using an old version of subctl, can you retry with 0.10.rc1 binary?

It was fixed recently as part of 0.10 development - https://github.com/submariner-io/submariner-operator/blob/16b97d57357d5355a0aea806475cda13174a7e87/pkg/subctl/cmd/diagnose/cni.go#L36

The troubleshooting command does not detect AKS default CNI - kubenet/calico and the supported CNI in the list doesnt mention calico .

Is it fair to say submariner with calico is not yet supported ?

We verified calico CNI and found it working after we explicitly configure the IPPools as documented here - https://submariner.io/operations/deployment/calico/ (Note: you can run subctl diagnose cni, it will verify the IPPools configuration in your clusters).

However, I remember someone in the community reported an issue saying that they are facing issues with Calico CNI even after the IPPools are configured. This is something that needs additional investigation.

To view this discussion on the web visit https://groups.google.com/d/msgid/submariner-users/CAEsNF5jPTKb3GtgCEdeCn-0c-c4sS%3DpELKx07eZxU31vnHpA3g%40mail.gmail.com.

[Monish R]

Hi Sridhar,

Thanks for the response, when trying to diagnose with 0.10.rc1 , it did show calico as supported CNI but the script failed to defect kubenet/calico CNI in the environment.

subctl diagnose cni

In submariner-operator-0.7.0 , we see that replicaset did not start after installation due to service account 'submariner-lighthouse' not found , once we created this sc the replicaset is up.

The daemonsets are still down and no logs to move further.

Regards,

R.Monish

[Sridhar Gaddam]

On Mon, Aug 16, 2021 at 11:58 PM Monish R <monishop...@gmail.com> wrote:

Hi Sridhar,

Thanks for the response, when trying to diagnose with 0.10.rc1 , it did show calico as supported CNI but the script failed to defect kubenet/calico CNI in the environment.

Currently, we detect the CNI as Calico with the presence of "calico-config" configMap. If your cluster is installed with Calico, do you think the name might be different?

 

subctl diagnose cni

In submariner-operator-0.7.0 , we see that replicaset did not start after installation due to service account 'submariner-lighthouse' not found , once we created this sc the replicaset is up.

The daemonsets are still down and no logs to move further.

Looks like both submariner-gateway and submariner-globalnet pods are not scheduled. Both of these DaemonSets have nodeSelector (submariner.io/gateway=true) configured.

Please check if you labelled at least one of your nodes as Gateway node and it has external-ip associated as mentioned in the pre-requisite section - https://submariner.io/getting-started/#prerequisites