starting submariner on IBM IKS clusters

Gopal Jayanti

unread,

Jun 30, 2021, 1:04:29 PM6/30/21

to submariner-users

HI

I was following the steps given in https://submariner.io/getting-started/quickstart/managed-kubernetes/gke/

and everything went smoothly until i got to this step...

This is not expected output, please guide me how to proceed.

kubectl version

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:20:10Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.11+IKS", GitCommit:"f9c13a7b4d87e89ac8abb7a30230f4c3d9272184", GitTreeState:"clean", BuildDate:"2021-05-14T21:38:40Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"}

subctl version

subctl version: v0.9.1

Thanks in advance

Gopal Jayanthi

subctl show all

Showing information for cluster "pspin/c0usfddd0f8jgeoojbk0":

Showing Network details

Discovered network details:

Network plugin: generic

Service CIDRs: [172.21.0.0/16]

Cluster CIDRs: []

Showing Endpoint details

No resources found.

Showing Connection details

No resources found.

Showing Gateway details

No resources found.

Showing version details

COMPONENT REPOSITORY VERSION

submariner quay.io/submariner 0.9.1

submariner-operator quay.io/submariner 0.9.1

service-discovery quay.io/submariner 0.9.1

Gopal Jayanti

unread,

Jul 1, 2021, 3:08:03 AM7/1/21

to submariner-users

I had the pod cidr range wrong when i corrected it to 172.17.0.0/18 , now i am getting this output, cluster b is not in the list, is it becasue pod cidr's are overlapping?

subctl show all

Showing information for cluster "pspin/*****************":

Showing Network details

Discovered network details:

Network plugin: generic

Service CIDRs: [172.21.0.0/16]

Cluster CIDRs: []

Showing Endpoint details

CLUSTER ID ENDPOINT IP PUBLIC IP CABLE DRIVER TYPE

cluster-a 10.240.64.11 **************** libreswan local

Showing Connection details

No resources found.

Showing Gateway details

NODE HA STATUS SUMMARY

kube-c0usfddd0f8jgeoojbk0-pspin active There are no connections

Showing version details

COMPONENT REPOSITORY VERSION

submariner quay.io/submariner 0.9.1

submariner-operator quay.io/submariner 0.9.1

service-discovery quay.io/submariner 0.9.1

Gopal Jayanti

unread,

Jul 1, 2021, 6:00:34 AM7/1/21

to submariner-users

subctl deploy-broker --globalnet true --globalnet-cidr-range 169.254.1.0/18

reran broker with globalnet but when i try to join cluster-a , i am geetting output saying globalnet not enabled

subctl join --globalnet-cidr 169.254.1.0/18 --clusterid cluster-a --servicecidr 172.21.0.0/16 broker-info.subm

* broker-info.subm says broker is at: https://c111.us-south.containers.cloud.ibm.com:32721

* There are 1 labeled nodes in the cluster:

- 10.240.64.11

⢄⡱ Discovering network details Discovered network details:

Network plugin: generic

Service CIDRs: [172.21.0.0/16]

Cluster CIDRs: []

✓ Discovering network details

? What's the Pod CIDR for your cluster? 172.17.0.0/18

✓ Validating Globalnet configurations

⚠ Globalnet is not enabled on Broker. Ignoring specified globalnet-cidr

✓ Discovering multi cluster details

✓ Deploying the Submariner operator

✓ Created Lighthouse service accounts and roles

✓ Creating SA for cluster

✓ Deploying Submariner

✓ Submariner is up and running

Sridhar Gaddam

unread,

Jul 1, 2021, 6:17:26 AM7/1/21

to Gopal Jayanti, submariner-users

Hello Gopal,

PSB inline.

On Thu, Jul 1, 2021 at 3:30 PM Gopal Jayanti <gopal....@opsmx.io> wrote:

subctl deploy-broker --globalnet true --globalnet-cidr-range 169.254.1.0/18

Was it a fresh cluster or did you run this command on an existing Broker which was initially deployed without Globalnet?

Looking at the error message from `subctl join` command, Globalnet does not seem to be enabled on the Broker, hence it's ignoring the globalnet flags that are supplied.

Basically when `subctl deploy-broker` command is executed with --globalnet enabled, it will store this in a configMap on the Broker, which you can see running the following command (on Broker Cluster).

kubectl describe cm submariner-globalnet-info -n submariner-k8s-broker

Can you tell us if the configMap data.globalnetEnabled is set to true or false?

Best Regards,

--Sridhar.

--
You received this message because you are subscribed to the Google Groups "submariner-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to submariner-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/submariner-users/168c8760-65ea-4742-be03-af8c97841ab5n%40googlegroups.com.

Gopal Jayanti

unread,

Jul 2, 2021, 7:59:21 AM7/2/21

to Sridhar Gaddam, submariner-users

Was it a fresh cluster or did you run this command on an existing Broker which was initially deployed without Globalnet?

i first deployed it without globalnet , then tried with globalnet

the cm has this data

Name: submariner-globalnet-info
Namespace: submariner-k8s-broker
Labels: component=submariner-globalnet
Annotations: <none>

Data
====
clusterinfo:
----
[]
globalnetEnabled:
----
false
Events: <none>

should i delete everything and restart?

I have some additional questions , do we need to run these to get submariner to work

wget https://raw.githubusercontent.com/sridhargaddam/k8sscripts/main/rp_filter_settings/update-rp-filter.sh wget https://raw.githubusercontent.com/sridhargaddam/k8sscripts/main/rp_filter_settings/configure-rp-filter.sh chmod +x update-rp-filter.sh chmod +x configure-rp-filter.sh gcloud container clusters get-credentials cluster-a --zone="europe-west3-a" ./configure-rp-filter.sh gcloud container clusters get-credentials cluster-b --zone="europe-west3-a" ./configure-rp-filter.sh

the pods created by this are crashing without any error message

also do you know the equivalent commands for these in ibm cloud?

gcloud compute firewall-rules create "allow-tcp-in" --allow=tcp \
--direction=IN --source-ranges=10.12.0.0/20,10.8.0.0/14,10.4.0.0/20,10.0.0.0/14

gcloud compute firewall-rules create "allow-tcp-out" --allow=tcp --direction=OUT \
--destination-ranges=10.12.0.0/20,10.8.0.0/14,10.4.0.0/20,10.0.0.0/14

gcloud compute firewall-rules create "udp-in-500" --allow=udp:500 --direction=IN
gcloud compute firewall-rules create "udp-in-4500" --allow=udp:4500 --direction=IN
gcloud compute firewall-rules create "udp-in-4800" --allow=udp:4800 --direction=IN

gcloud compute firewall-rules create "udp-out-500" --allow=udp:500 --direction=OUT
gcloud compute firewall-rules create "udp-out-4500" --allow=udp:4500 --direction=OUT
gcloud compute firewall-rules create "udp-out-4800" --allow=udp:4800 --direction=OUT

Thanks in advance

Gopal Jayanthi

Miguel Angel Ajo

unread,

Jul 2, 2021, 8:07:49 AM7/2/21

to Gopal Jayanti, Sridhar Gaddam, submariner-users

On Fri, Jul 2, 2021 at 1:59 PM Gopal Jayanti <gopal....@opsmx.io> wrote:

Was it a fresh cluster or did you run this command on an existing Broker which was initially deployed without Globalnet?
i first deployed it without globalnet , then tried with globalnet

the cm has this data

Name: submariner-globalnet-info
Namespace: submariner-k8s-broker
Labels: component=submariner-globalnet
Annotations: <none>

Data
====
clusterinfo:
----
[]
globalnetEnabled:
----
false
Events: <none>

should i delete everything and restart?

Probably a workaround could be deleting this configmap, and re-deploying the broker with globalnet enabled. Currently we don't support switching from globalnet to no globalnet or the other way around because it could be disruptive on the dataplane. But at least I believe we may get an error condition.

I have some additional questions , do we need to run these to get submariner to work

wget https://raw.githubusercontent.com/sridhargaddam/k8sscripts/main/rp_filter_settings/update-rp-filter.sh wget https://raw.githubusercontent.com/sridhargaddam/k8sscripts/main/rp_filter_settings/configure-rp-filter.sh chmod +x update-rp-filter.sh chmod +x configure-rp-filter.sh gcloud container clusters get-credentials cluster-a --zone="europe-west3-a" ./configure-rp-filter.sh gcloud container clusters get-credentials cluster-b --zone="europe-west3-a" ./configure-rp-filter.sh

the pods created by this are crashing without any error message

That really depends on how the IBM IKS works at the network level, we got access recently to the IBM cloud, but not sure of IKS. May be we can have a look if you join us on slack.

also do you know the equivalent commands for these in ibm cloud?

gcloud compute firewall-rules create "allow-tcp-in" --allow=tcp \
--direction=IN --source-ranges=10.12.0.0/20,10.8.0.0/14,10.4.0.0/20,10.0.0.0/14

gcloud compute firewall-rules create "allow-tcp-out" --allow=tcp --direction=OUT \
--destination-ranges=10.12.0.0/20,10.8.0.0/14,10.4.0.0/20,10.0.0.0/14

Those two are probably not necessary ^, again, depends on how IKS is implemented.

gcloud compute firewall-rules create "udp-in-500" --allow=udp:500 --direction=IN
gcloud compute firewall-rules create "udp-in-4500" --allow=udp:4500 --direction=IN
gcloud compute firewall-rules create "udp-in-4800" --allow=udp:4800 --direction=IN

gcloud compute firewall-rules create "udp-out-500" --allow=udp:500 --direction=OUT
gcloud compute firewall-rules create "udp-out-4500" --allow=udp:4500 --direction=OUT
gcloud compute firewall-rules create "udp-out-4800" --allow=udp:4800 --direction=OUT

Those ones are probably necessary, but I don't know the ibm cloud equivalent yet, is there a cmdline tool/how does the

console look like?

For the 4500 port, the first thing you could see if that port needs to be open, is that connections will

remain on a "connecting" state.

Thanks in advance

Sorry to not be 100% helpful here, this is unexplored territory to us, but let's make it work.

To view this discussion on the web visit https://groups.google.com/d/msgid/submariner-users/CAPXpjtiQiNKouuGBk3LJnLpoOfW%2B1cVcDNBBWEvLBSGAmHjW8w%40mail.gmail.com.

--

Miguel Ángel Ajo @mangel_ajo

OpenShift / Kubernetes / Multi-cluster Networking team.

ex OSP / Networking DFG, OVN Squad Engineering

Gopal Jayanti

unread,

Jul 2, 2021, 8:10:40 AM7/2/21

to Miguel Angel Ajo, Sridhar Gaddam, submariner-users

Thanks for the feedback. I would like to join you on slack.

Miguel Angel Ajo

unread,

Jul 2, 2021, 9:20:59 AM7/2/21

to Gopal Jayanti, Sridhar Gaddam, submariner-users

You can find us here on the k8s slack:

https://kubernetes.slack.com/archives/C010RJV694M

If you don't have an account, you can get an invite here: https://slack.k8s.io/

Gopal Jayanti

unread,

Jul 3, 2021, 2:12:27 PM7/3/21

to Miguel Angel Ajo, Sridhar Gaddam, submariner-users

HI Miguel/Sridhar

Thanks, I joined the slack channel.

Meanwhile, i created two more clusters with non-overlapping cidrs in IBM cloud ( so i dont have to use global net) and I am facing the same issue.

subctl show all

Showing information for cluster "submariner-c-test-cluster/c3fvuprd0vobcoseot80":

Showing Network details
Discovered network details:
Network plugin: generic
Service CIDRs: [172.21.0.0/16]
Cluster CIDRs: []

Showing Endpoint details
CLUSTER ID ENDPOINT IP PUBLIC IP CABLE DRIVER TYPE

c3fvuprd0vobcoseot80 10.240.128.4 xxxxxxxxxxxx libreswan local
c3g1045d0j7fl990v4q0 10.240.64.12 xxxxxxxxxxxx libreswan remote

Showing Connection details
GATEWAY CLUSTER REMOTE IP NAT CABLE DRIVER SUBNETS STATUS RTT avg.
kube-c3g1045d0j7fl990v4q0-subma c3g1045d0j7fl990v4q0 xxxxxxxxxx yes libreswan 172.22.0.0/16, 172.17.0.0/18 connecting 0s

Showing Gateway details
NODE HA STATUS SUMMARY

kube-c3fvuprd0vobcoseot80-subma active 0 connections out of 1 are established

Showing version details
COMPONENT REPOSITORY VERSION
submariner quay.io/submariner 0.9.1
submariner-operator quay.io/submariner 0.9.1
service-discovery quay.io/submariner 0.9.1

I am attaching the logs from suctl verify and subctl gather

gather-c.log

verify.log

gather.zip

Sridhar Gaddam

unread,

Jul 4, 2021, 7:54:31 AM7/4/21

to Gopal Jayanti, Miguel Angel Ajo, submariner-users

Hello Gopal,

Looking at the logs, it appears like you are using Calico CNI which requires some additional configuration as explained here - https://submariner.io/operations/deployment/calico/

Along with this, I think the firewall configuration is not allowing tunnels to be setup hence IPsec tunnels are not connected.

Please check if you opened the firewall ports as described in the pre-requisites section.

https://submariner.io/getting-started/#prerequisites

You can run `subctl diagnose all` as well as `subctl diagnose firewall tunnel` which will tell you if there are any issues with the cluster configuration.

Best Regards,

--Sridhar.

Gopal Jayanti

unread,

Jul 4, 2021, 11:51:14 AM7/4/21

to submariner-users

HI Sridhar

We have followed the instruction to the word for noth the calico steps and firewall steps. Quite sure the issue is elsewhere.

Thanks

Gopal Jayanthi

Sridhar Gaddam

unread,

Jul 4, 2021, 1:33:19 PM7/4/21

to Gopal Jayanti, submariner-users

On Sun, Jul 4, 2021 at 9:21 PM Gopal Jayanti <gopal....@opsmx.io> wrote:

HI Sridhar

We have followed the instruction to the word for noth the calico steps and firewall steps. Quite sure the issue is elsewhere.

From the logs you shared earlier, I could see Calico inserting its iptable rules ahead of Submariner rules which generally happens when we do not program the necessary Calico IPPools.

Anyways, can you pls share the output of the following commands?

subctl diagnose all

subctl diagnose firewall tunnel

Best Regards,

--Sridhar.

To view this discussion on the web visit https://groups.google.com/d/msgid/submariner-users/166046bb-c566-4329-bb9d-09d99e315816n%40googlegroups.com.

Gopal Jayanti

unread,

Jul 5, 2021, 2:12:54 AM7/5/21

to Sridhar Gaddam, submariner-users

subctl diagnose all
✓ Checking Submariner support for the Kubernetes version used in cluster "submariner-c-test-cluster/c3fvuprd0vobcoseot80"
✓ The Kubernetes version meets Submariner's requirements

✓ Retrieving Submariner resource from "submariner-c-test-cluster/c3fvuprd0vobcoseot80"

✓ Checking Submariner support for the CNI network plugin in cluster "submariner-c-test-cluster/c3fvuprd0vobcoseot80"
✓ The detected CNI network plugin ("generic") is supported by Submariner.
✓ Calico CNI detected, verifying if the Submariner IPPool pre-requisites are configured.

✗ Checking Gateway connections in cluster "submariner-c-test-cluster/c3fvuprd0vobcoseot80"
✗ Connection to cluster "c3g1045d0j7fl990v4q0" is in progress

Skipping tunnel firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall tunnel" command manually.

subctl diagnose firewall tunnel /home/ubuntu/submariner/cluster-c.config /home/ubuntu/submariner/cluster-d.config
⚠ Checking if tunnels can be setup on Gateway node of cluster "c3fvuprd0vobcoseot80".
⚠ Could not find the active Gateway nodeName in local cluster

k describe gateways.submariner.io -n submariner-operator kube-c3fvuprd0vobcoseot80-submarinerc-submari-00000146
Name: kube-c3fvuprd0vobcoseot80-submarinerc-submari-00000146
Namespace: submariner-operator
Labels: <none>
Annotations: update-timestamp: 1625435263
API Version: submariner.io/v1
Kind: Gateway
Metadata:
Creation Timestamp: 2021-07-04T21:47:33Z
Generation: 3
Managed Fields:
API Version: submariner.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:update-timestamp:
f:status:
.:
f:connections:
f:haStatus:
f:localEndpoint:
.:
f:backend:
f:backend_config:
.:
f:preferred-server:
f:cable_name:
f:cluster_id:
f:healthCheckIP:
f:hostname:
f:nat_enabled:
f:private_ip:
f:public_ip:
f:subnets:
f:statusFailure:
f:version:
Manager: submariner-gateway
Operation: Update
Time: 2021-07-04T21:47:33Z
Resource Version: 302733
UID: eef3a1b7-29d7-4628-abde-72520696fec0
Status:
Connections:
Endpoint:
Backend: libreswan
backend_config:
Preferred - Server: false
cable_name: submariner-cable-c3g1045d0j7fl990v4q0-10-240-64-12
cluster_id: c3g1045d0j7fl990v4q0
Health Check IP: 172.17.35.0
Hostname: kube-c3g1045d0j7fl990v4q0-submarinerd-default-000001fb
nat_enabled: true
private_ip: 10.240.64.12
public_ip: 150.239.164.46
Subnets:
172.22.0.0/16
172.17.0.0/18
Latency RTT:
Average: 0s
Last: 0s
Max: 0s
Min: 0s
Std Dev: 0s
Status: connecting
Status Message:
Using IP: 150.239.164.46
Using NAT: true
Ha Status: active
Local Endpoint:
Backend: libreswan
backend_config:
Preferred - Server: false
cable_name: submariner-cable-c3fvuprd0vobcoseot80-10-240-128-4
cluster_id: c3fvuprd0vobcoseot80
Health Check IP: 172.17.111.0
Hostname: kube-c3fvuprd0vobcoseot80-submarinerc-submari-00000146
nat_enabled: true
private_ip: 10.240.128.4
public_ip: 52.118.79.113
Subnets:
172.21.0.0/16
172.17.64.0/18
Status Failure:
Version: release-0.9-c0e2e61
Events: <none>

In GKE cluster the hostname was coming as the actual nodename. IN IBM cluster it is not name but ibm-cloud.kubernetes.io/worker-id

Sridhar Gaddam

unread,

Jul 7, 2021, 4:53:36 AM7/7/21

to Gopal Jayanti, submariner-users

On Mon, Jul 5, 2021 at 11:43 AM Gopal Jayanti <gopal....@opsmx.io> wrote:

subctl diagnose all
✓ Checking Submariner support for the Kubernetes version used in cluster "submariner-c-test-cluster/c3fvuprd0vobcoseot80"
✓ The Kubernetes version meets Submariner's requirements

✓ Retrieving Submariner resource from "submariner-c-test-cluster/c3fvuprd0vobcoseot80"

✓ Checking Submariner support for the CNI network plugin in cluster "submariner-c-test-cluster/c3fvuprd0vobcoseot80"
✓ The detected CNI network plugin ("generic") is supported by Submariner.
✓ Calico CNI detected, verifying if the Submariner IPPool pre-requisites are configured.

Okay, the CalicoIPPools seem to be properly configured.

We have to investigate why the tunnels/connections are not getting established (this has nothing to do with the Calico IPPools and is most likely related to some firewall/external-ip configuration for the Gateway nodes)

✗ Checking Gateway connections in cluster "submariner-c-test-cluster/c3fvuprd0vobcoseot80"
✗ Connection to cluster "c3g1045d0j7fl990v4q0" is in progress

Ideally, I was expecting `subctl diagnose all` to continue validating the remaining steps like deployment, kubeproxy-mode etc, but it looks like after there was a failure it stopped validating the remaining.

I reported the following issue to track it - https://github.com/submariner-io/submariner-operator/issues/1472

Skipping tunnel firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall tunnel" command manually.

subctl diagnose firewall tunnel /home/ubuntu/submariner/cluster-c.config /home/ubuntu/submariner/cluster-d.config
⚠ Checking if tunnels can be setup on Gateway node of cluster "c3fvuprd0vobcoseot80".
⚠ Could not find the active Gateway nodeName in local cluster

Because there is a mismatch in the hostname vs the nodenames on IBMCluster, subctl diagnose could not validate the firewall configuration for IPsec tunnels.

We will have to handle this in subctl diagnose tool and the following issue is reported for that - https://github.com/submariner-io/submariner-operator/issues/1471

As Miguel mentioned earlier, we have not tested much with IBMCloud and this needs some investigation.

So, Gopal, as discussed with you over slack, currently we are a bit busy getting the last few bits before we make the rc0 release and we might get some bandwidth next week to look into this.

Sridhar Gaddam

unread,

Jul 7, 2021, 5:10:32 AM7/7/21

to Gopal Jayanti, submariner-users

On Wed, Jul 7, 2021 at 2:23 PM Sridhar Gaddam <sga...@redhat.com> wrote:

On Mon, Jul 5, 2021 at 11:43 AM Gopal Jayanti <gopal....@opsmx.io> wrote:
subctl diagnose all
✓ Checking Submariner support for the Kubernetes version used in cluster "submariner-c-test-cluster/c3fvuprd0vobcoseot80"
✓ The Kubernetes version meets Submariner's requirements

✓ Retrieving Submariner resource from "submariner-c-test-cluster/c3fvuprd0vobcoseot80"

✓ Checking Submariner support for the CNI network plugin in cluster "submariner-c-test-cluster/c3fvuprd0vobcoseot80"
✓ The detected CNI network plugin ("generic") is supported by Submariner.
✓ Calico CNI detected, verifying if the Submariner IPPool pre-requisites are configured.
Okay, the CalicoIPPools seem to be properly configured.
We have to investigate why the tunnels/connections are not getting established (this has nothing to do with the Calico IPPools and is most likely related to some firewall/external-ip configuration for the Gateway nodes)

✗ Checking Gateway connections in cluster "submariner-c-test-cluster/c3fvuprd0vobcoseot80"
✗ Connection to cluster "c3g1045d0j7fl990v4q0" is in progress
Ideally, I was expecting `subctl diagnose all` to continue validating the remaining steps like deployment, kubeproxy-mode etc, but it looks like after there was a failure it stopped validating the remaining.
I reported the following issue to track it - https://github.com/submariner-io/submariner-operator/issues/1472

Gopal, may I know the version of subctl that you were using while running subctl diagnose command? You can get the info using the following command `subctl version`

Gopal Jayanti

unread,

Jul 7, 2021, 5:46:00 AM7/7/21

to Sridhar Gaddam, submariner-users

Sure, here it is.

0.9.1

Miguel Angel Ajo

unread,

Jul 7, 2021, 5:46:47 AM7/7/21

to Sridhar Gaddam, Gopal Jayanti, submariner-users

I wanted to have a session with Gopal today to troubleshoot this, but it has turned out into a very busy day,

we want to retry on friday.

Based on the last info I had from him (the tunnels had established, etc.) I suspect that at this point it could

be the VPC firewall filtering the remote CIDRs on the gateway->worker transit (which we don't encapsulate).

To view this discussion on the web visit https://groups.google.com/d/msgid/submariner-users/CAOW2eyzvzrBsd4QvRF-KiYc9mKnE_ZCQ66Vq4igWxMF%3DgjHGPA%40mail.gmail.com.

Sridhar Gaddam

unread,

Jul 7, 2021, 7:47:06 AM7/7/21

to Gopal Jayanti, submariner-users

On Wed, Jul 7, 2021 at 3:16 PM Gopal Jayanti <gopal....@opsmx.io> wrote:

Sure, here it is.
0.9.1

Okay, thanks. Issue#1472 was already fixed a few days back and will be part of the official 0.10 release (or devel in the meantime).

https://github.com/submariner-io/submariner-operator/commit/8eb5ecbea1d541037145ae54ba51c9711d194ecd

Reply all

Reply to author

Forward