submariner in azure AKS

86 views
Skip to first unread message

Monish R

unread,
Aug 10, 2021, 2:11:41 PM8/10/21
to submariner-users
Hi all,
we are trying out submariner in azure aks environment in two clusters, after following the helm installation steps provided in link 


Note: our Env has,
CNI : kubenet / calico . 
We have not yet executed the post installation steps specific to calico CNI. assume that gateway should be up regardless.

we observe the deployed resources are not properly up in both cluster.

submariner-operator          pod/submariner-gateway-gwsv9                                        0/1     CrashLoopBackOff             38         170m
submariner-operator          deployment.apps/submariner-lighthouse-agent                       0/1     0            0           170m
submariner-operator          deployment.apps/submariner-lighthouse-coredns                     0/2     0            0           170m

Checking the logs on the submariner gateway :

I0810 17:56:36.097872       1 main.go:71] Starting the submariner gateway engine
W0810 17:56:36.098404       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0810 17:56:36.098649       1 main.go:99] Creating the cable engine
I0810 17:56:36.311212       1 syncer.go:51] CableEngine syncer started
I0810 17:56:36.311821       1 main.go:263] Gateway leader election config values: main.leaderConfig{LeaseDuration:10, RenewDeadline:5, RetryPeriod:2}
I0810 17:56:36.311991       1 main.go:280] Using namespace "submariner-operator" for the leader election lock
I0810 17:56:36.312011       1 leaderelection.go:242] attempting to acquire leader lease  submariner-operator/submariner-engine-lock...
I0810 17:56:36.345107       1 leaderelection.go:252] successfully acquired lease submariner-operator/submariner-engine-lock
I0810 17:56:36.345144       1 main.go:133] Creating the tunnel controller
I0810 17:56:36.345143       1 event.go:281] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"submariner-operator", Name:"submariner-engine-lock", UID:"5730b78c-c596-43d2-b1a5-6212f62ec983", APIVersion:"v1", ResourceVersion:"1074527", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' aks-devfecpool-17806234-vmss000000-submariner-engine became leader
I0810 17:56:36.345184       1 main.go:141] Creating the kubernetes central datastore
I0810 17:56:36.345211       1 kubernetes.go:60] Rendered API server host: "https://13.XX.XX.XXX:443"
F0810 17:56:36.388633       1 main.go:313] Error creating the kubernetes datastore: error building submariner clientset: Get "https://13.XX.XX.XXX:443/apis/submariner.io/v1/namespaces/submariner-k8s-broker/endpoints": x509: certificate is valid for 10.XX.X.X, not 13.XX.XX.XXX

Any pointers in resolving this error is appreciated.

Regards,
R.Monish


Vishal Thapar

unread,
Aug 11, 2021, 8:43:56 AM8/11/21
to Monish R, submariner-users
Hi Monish,

This looks like the API server endpoint given is 13.x.x.x while certificate is for 10.x.x.x  Can you can get API endpoint to use from your KUBECONFIG and try using that instead?

Regards,
Vishal.

--
You received this message because you are subscribed to the Google Groups "submariner-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to submariner-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/submariner-users/c168ca8c-085b-4e44-946e-78f04e63cc13n%40googlegroups.com.

Monish R

unread,
Aug 11, 2021, 6:00:27 PM8/11/21
to Vishal Thapar, submariner-users
Hi Visha Thapar,
Thanks for the response.
As you have pointed out 13.x.x.x IP is the API endpoint from KUBECONFIG and  10.x.x.x  mentioned in the log is the service CIDR IP.

In which configuration do we need to make a change ?

Also when we ran : 

$ subctl diagnose all

✓ Checking Submariner support for the Kubernetes version used in cluster "cluster-B"
 ✓ The Kubernetes version meets Submariner's requirements

 ✓ Retrieving Submariner resource from " cluster-B "

 ✗ Checking Submariner support for the CNI network plugin in cluster " cluster-B "
 ✗ The detected CNI network plugin ("") is not supported by Submariner. Supported network plugins: [generic canal-flannel weave-net OpenShiftSDN OVNKubernetes]

The troubleshooting command does not detect AKS default CNI - kubenet/calico and the supported CNI in the list doesnt mention calico .

Is it fair to say submariner with calico is not yet supported ?

Regards,
R.Monish

Vishal Thapar

unread,
Aug 12, 2021, 3:45:10 AM8/12/21
to Monish R, submariner-users
Hi Monish,

You will have to redeploy and modify this env variable [taken from helm install guide]

export SUBMARINER_BROKER_URL=$(kubectl -n default get endpoints kubernetes \ -o jsonpath="{.subsets[0].addresses[0].ip}:{.subsets[0].ports[?(@.name=='https')].port}")

IIRC, when you get it from endpoints it will return the 10.x.x.x IP. We should probably change this to pick IP from kubeconfig.

Let me know if this helps. You can reach out to us in our slack https://submariner.io/community/getting-help/#slack 

Regards,
Vishal.

Sridhar Gaddam

unread,
Aug 16, 2021, 3:18:21 AM8/16/21
to Monish R, Vishal Thapar, submariner-users
Hello Monish,

PSB inline.

On Thu, Aug 12, 2021 at 3:30 AM Monish R <monishop...@gmail.com> wrote:
Hi Visha Thapar,
Thanks for the response.
As you have pointed out 13.x.x.x IP is the API endpoint from KUBECONFIG and  10.x.x.x  mentioned in the log is the service CIDR IP.

In which configuration do we need to make a change ?

Also when we ran : 

$ subctl diagnose all

✓ Checking Submariner support for the Kubernetes version used in cluster "cluster-B"
 ✓ The Kubernetes version meets Submariner's requirements

 ✓ Retrieving Submariner resource from " cluster-B "

 ✗ Checking Submariner support for the CNI network plugin in cluster " cluster-B "
 ✗ The detected CNI network plugin ("") is not supported by Submariner. Supported network plugins: [generic canal-flannel weave-net OpenShiftSDN OVNKubernetes]
If you are using an old version of subctl, can you retry with 0.10.rc1 binary?


The troubleshooting command does not detect AKS default CNI - kubenet/calico and the supported CNI in the list doesnt mention calico .

Is it fair to say submariner with calico is not yet supported ?
We verified calico CNI and found it working after we explicitly configure the IPPools as documented here - https://submariner.io/operations/deployment/calico/ (Note: you can run subctl diagnose cni, it will verify the IPPools configuration in your clusters).
However, I remember someone in the community reported an issue saying that they are facing issues with Calico CNI even after the IPPools are configured. This is something that needs additional investigation.

Monish R

unread,
Aug 16, 2021, 2:28:28 PM8/16/21
to Sridhar Gaddam, Vishal Thapar, submariner-users
Hi Sridhar,
Thanks for the response, when trying to diagnose with 0.10.rc1 , it did show calico as supported CNI but the script failed to defect kubenet/calico CNI in the environment.

subctl diagnose cni
image.png

In submariner-operator-0.7.0 , we see that replicaset did not start after installation due to service account 'submariner-lighthouse' not found , once we created this sc the replicaset is up.

The daemonsets are still down and no logs to move further.

image.png

Regards,
R.Monish

Sridhar Gaddam

unread,
Aug 17, 2021, 3:15:02 AM8/17/21
to Monish R, Vishal Thapar, submariner-users
On Mon, Aug 16, 2021 at 11:58 PM Monish R <monishop...@gmail.com> wrote:
Hi Sridhar,
Thanks for the response, when trying to diagnose with 0.10.rc1 , it did show calico as supported CNI but the script failed to defect kubenet/calico CNI in the environment.

Currently, we detect the CNI as Calico with the presence of "calico-config" configMap. If your cluster is installed with Calico, do you think the name might be different?
 
subctl diagnose cni
image.png

In submariner-operator-0.7.0 , we see that replicaset did not start after installation due to service account 'submariner-lighthouse' not found , once we created this sc the replicaset is up.

The daemonsets are still down and no logs to move further.
Looks like both submariner-gateway and submariner-globalnet pods are not scheduled. Both of these DaemonSets have nodeSelector (submariner.io/gateway=true) configured.
Please check if you labelled at least one of your nodes as Gateway node and it has external-ip associated as mentioned in the pre-requisite section - https://submariner.io/getting-started/#prerequisites
Reply all
Reply to author
Forward
0 new messages