Re: [sdfabric-announce] Installation Instructions for GTP on EdgeCore

37 views
Skip to first unread message

Chan, Charles

unread,
Jun 28, 2022, 2:38:53 AM6/28/22
to David Lake, sdfabr...@opennetworking.org

(Moving to sdfabric-dev, which is more suitable for this discussion)

Hi David,

It is recommended to deploy using Kubernetes and Helm. You can download the SD-Fabric umbrella helm chart here.
The latest SD-Fabric release version is 1.1.2. You can find the version of each sub-components here.

Thanks.

Intel

Charles Chan, PhD (he/him)

Cloud Software Development Engineer

Software Product Incubation | Network and Edge Group

Intel Corporation  |  intel.com

 

On 6/27/22, 04:57, "'David Lake' via SDFABRIC-Announce" <sdfabric...@opennetworking.org> wrote:


Hello

 

I'm trying to follow the deployment guide for SD-Fabric.

 

I'm building a very simple environment - one EdgeCore switch running latest Stratum to do GTP encap/decap as a hardware UPF in a a Free5GC network.

 

Stratum is running on the switch but I don't understand what I need to do next. I need to install ONOS but I don't know which version and then I don't understand how to install the PFCP agent that is discussed in the guide.

 

Is 'SD-Fabric' a package of ONOS/PFCP/DBUF that I can simply pull a Docker container or do I have to manually install ONS first?  In which case, what version is tested?

 

Many thanks

 

David

--
You received this message because you are subscribed to the Google Groups "SDFABRIC-Announce" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sdfabric-annou...@opennetworking.org.
To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/sdfabric-announce/04f5bd33-9595-4383-999b-5c1c092a20fan%40opennetworking.org.

David Lake

unread,
Jun 28, 2022, 5:42:04 AM6/28/22
to Chan, Charles, sdfabr...@opennetworking.org

Hi Charlie

 

Thank you but I’m having immense difficulty understanding the deployment procedures and the order of operations – there seems to be some circular syntax on deployment page…

 

  1. With reference to https://docs.sd-fabric.org/master/deployment.html , ‘Step 2: Configure switches as special Kubernetes nodes’ immediately after the configuration of the switch.  The discussion appears centred around configuration of the switch as it follows from the terminal session in section 1 but I am unsure is the two items referred to (‘Label’ and ‘Taint’) are shell commands on the switch or within part of the configuration of Kubernetes on the switch.  The documentation states that I should set-up ‘Label’ and ‘Taint’ but it gives no details of these items or where they should be set?  Ditto with ‘NodeSelector’ and ‘Toleration’ – is this a local configuration to Kubernetes on the switch (in which case where/how) or do I have to include this on my Kubernetes Control Plane somewhere?
  2. The ‘example of five nodes Kubernetes’ – is this executed on the switch or somewhere else?  As far as I can see, at this point in the configuration we haven’t actually deployed anything relating to the switch layout (I believe that is in Step 5 of this page but I’m not sure?).
  3. Step 3 points to a separate page that discusses ONOS Network Configuration; this page details various options but does not mention where these configuration files should be located?
  4. Step 4 similarly points to configuration relevant to Stratum but again does not give the name nor location of the created configuration file?
  5. Step 5 – this is the most confusing part!  I’m very unclear about the ‘Prerequisites’ – there is mention of a ‘Step 5’ pre-requisite but I only see 4? https://gerrit.opencord.org/plugins/gitiles/sdfabric-helm-charts/+/HEAD/sdfabric/README.md

 

I’m also assuming that the whole point of the SD-Fabric Helm chart is that I no longer need to manually install/configure Stratum on the switches, ONOS, PFCP, DBUF on the controller node – am I correct in this assumption or is there something I’ve missed?

 

Many thanks for your assistance

 

David

David Lake

unread,
Jun 28, 2022, 6:03:05 AM6/28/22
to Chan, Charles, sdfabr...@opennetworking.org

One other question – there is no step whereby the switches join the Kubernetes cluster as a regular node would; do I need to do this (if so, how?) or is this part of the SD-Fabric implementation?

 

Thanks again!


David

--
You received this message because you are subscribed to the Google Groups "SDFABRIC-Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sdfabric-dev...@opennetworking.org.
To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/sdfabric-dev/DB7PR06MB47920425B2E60A2852AA169BB5B89%40DB7PR06MB4792.eurprd06.prod.outlook.com.

Chan, Charles

unread,
Jun 28, 2022, 1:29:58 PM6/28/22
to David Lake, sdfabr...@opennetworking.org

You can find more information here: https://docs.sd-fabric.org/master/deployment.html


Step 2: Configure switches as special Kubernetes nodes

Thanks.

 

Intel

Charles Chan, PhD (he/him)

Cloud Software Development Engineer

Software Product Incubation | Network and Edge Group

Intel Corporation  |  intel.com

 

On 6/28/22, 03:04, "David Lake" <d.l...@surrey.ac.uk> wrote:

 

One other question – there is no step whereby the switches join the Kubernetes cluster as a regular node would; do I need to do this (if so, how?) or is this part of the SD-Fabric implementation?

 

Thanks again!


David

 

From: 'David Lake' via SDFABRIC-Dev <sdfabr...@opennetworking.org>
Date: Tuesday, 28 June 2022 at 10:42
To: Chan, Charles <charle...@intel.com>, sdfabr...@opennetworking.org <sdfabr...@opennetworking.org>
Subject: [sdfabric-dev] Re: [sdfabric-announce] Installation Instructions for GTP on EdgeCore

Hi Charlie

 

Thank you but I’m having immense difficulty understanding the deployment procedures and the order of operations – there seems to be some circular syntax on deployment page…

 

1)       With reference to https://docs.sd-fabric.org/master/deployment.html , ‘Step 2: Configure switches as special Kubernetes nodes’ immediately after the configuration of the switch.  The discussion appears centred around configuration of the switch as it follows from the terminal session in section 1 but I am unsure is the two items referred to (‘Label’ and ‘Taint’) are shell commands on the switch or within part of the configuration of Kubernetes on the switch.  The documentation states that I should set-up ‘Label’ and ‘Taint’ but it gives no details of these items or where they should be set?  Ditto with ‘NodeSelector’ and ‘Toleration’ – is this a local configuration to Kubernetes on the switch (in which case where/how) or do I have to include this on my Kubernetes Control Plane somewhere?

2)       The ‘example of five nodes Kubernetes’ – is this executed on the switch or somewhere else?  As far as I can see, at this point in the configuration we haven’t actually deployed anything relating to the switch layout (I believe that is in Step 5 of this page but I’m not sure?).

3)       Step 3 points to a separate page that discusses ONOS Network Configuration; this page details various options but does not mention where these configuration files should be located?

4)       Step 4 similarly points to configuration relevant to Stratum but again does not give the name nor location of the created configuration file?

5)       Step 5 – this is the most confusing part!  I’m very unclear about the ‘Prerequisites’ – there is mention of a ‘Step 5’ pre-requisite but I only see 4? https://gerrit.opencord.org/plugins/gitiles/sdfabric-helm-charts/+/HEAD/sdfabric/README.md

David Lake

unread,
Jun 28, 2022, 4:07:43 PM6/28/22
to Chan, Charles, sdfabr...@opennetworking.org

Hi Charles

 

Apologies, but I don’t understand.

I have installed the ONL version on the switch but there is no Kubernetes on it.

 

How do I add the switches to the Kubernetes control plane?

 

Thanks

David Lake

unread,
Jun 28, 2022, 6:40:10 PM6/28/22
to Chan, Charles, sdfabr...@opennetworking.org

Hi

 

Just to follow up on this.

 

I’ve eventually worked out how to download the Helm charts – the URL in the documentation is incorrect (or at least requires an SSH key) and should be git clone https://gerrit.opencord.org/sdfabric-helm-charts

 

Once the Helm charts were on my Control Plane node I looked at the README.md which states:

 

There are several preparation work that need to be done before deploying SD-Fabric.

1. All switches have been provisioned with Docker and Kubernetes

2. All switches have been configured as special Kubernetes nodes

3. Prepare ONOS network config

4. Prepare Stratum chassis config for each switch

 

 

The switch has been installed using the same version of ONL as in the documentation but there is no Kuberenetes installed on it and ‘apt-get install kubeadm kubectl’ fails to find a candidate in the repo.

 

So I am unsure whether I need to install Kubernetes manually or from another repo.

 

Please advise how I should install Kubernetes on the switches.

 

Many thanks

David Lake

unread,
Jun 29, 2022, 9:31:42 AM6/29/22
to Chan, Charles, sdfabr...@opennetworking.org

Hi all

 

OK – I’ve managed to work out how to install Kubernetes on the EdgeCore Switch but I haven’t yet tried to join the Kubernetes cluster.

 

The instructions say that I need to ‘… set up Label to all switch node…’ and ‘… Taint to all switch node…’

 

I don’t understand WHERE this is set?  Is that on the Kubernetes node (i.e. the switch) in which case how and where is that done?

 

Or is it on the Kubernetes control plane – again how and where is that done?

 

Thanks

Chan, Charles

unread,
Jun 30, 2022, 1:39:33 AM6/30/22
to David Lake, sdfabr...@opennetworking.org

We use Rancher to provision K8s, but any existing solutions that works on x86 servers will work for the switches theoretically. After all, the switch is yet another x86 devices running Linux. I’d suggest go with whatever K8s management tools you are familiar with.

It’s the same for the taints and labels. There is nothing different from operating typical K8s cluster.

K8s configuration and operation are beyond the scope of the SD-Fabric, but there are a lot of resources on the Internet. For example:
https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/


Thanks.

Intel

Charles Chan, PhD (he/him)

Cloud Software Development Engineer

Software Product Incubation | Network and Edge Group

Intel Corporation  |  intel.com

 

David Lake

unread,
Jun 30, 2022, 5:39:37 PM6/30/22
to Chan, Charles, sdfabr...@opennetworking.org

The problem I’m having is much more basic than Rancher – I can’t even get K8S to run on the EdgeCore in the way that your documentation says it does.

 

I’ve downloaded and installed the exact version of ONL that the website states and there is no Kubernetes on there and no instructions on your website as to what to do.

 

Given that K8S on the EdgeCore is a fundamental part of this project, all I want to do is to follow the steps your team has taken to make this work.  I’ve tried to download the latest K8S to the switch and hit a number of errors.

 

Can you tell me what version of K8S you have running on your EdgeCore switches and how you installed it? 

 

Thank you


David

David Lake

unread,
Jul 1, 2022, 7:26:08 AM7/1/22
to Chan, Charles, sdfabr...@opennetworking.org

Success!

 

I’ve finally been able to install Kubernetes on the ONL system.  The latest K8S is 1.24.2 but this will not run on Debian Stretch with Docker 20.x.x which is the version ONL is based on so I went back to Docker 19.03.15 and K8S 1.18.20.

 

As my CP node is Ubuntu 20.04, I’ve had to force installation of older packages:

 

                apt-get install containerd=1.3.3-0ubuntu2

apt-get install docker.io=19.03.8-0ubuntu1

 

On both the switch and the CP node I installed:

 

                apt install kubectl=1.18.20-00 kubelet=1.18.20-00 kubernetes-cni cri-tools=1.19.0-00 kubeadm=1.18.20-00

 

I also found that I needed to change the cgroups driver to systemd for Docker on the CP node in /etc/docker/daemon.json:

{

  "exec-opts": ["native.cgroupdriver=systemd"]

}

 

I’m using Flannel networking but found that the initnetdev script was missing in ONL – I added it from here https://github.com/opennetworklinux/ONL/blob/master/builds/swi/all/shared/sbin/initnetdev

 

Once that was installed, I ran
                     kubeadm init --pod-network-cidr 10.244.0.0/16

.. on the Control Plane node and the corresponding kubeadm on the switch.  I then installed Flannel:

 

                kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

 

I’m now in the process of labelling and tainting the switch before I attempt to deploy SD-Fabric.

 

I hope this helps anyone else going through the same issues.   If ONL is bumped to Debian 10 at any point, then it would be possible to run K8S 1.24 and newer Docker so these steps will change.

 

David

 


Date: Thursday, 30 June 2022 at 06:39

Hung-Wei Chiu

unread,
Jul 5, 2022, 11:53:15 PM7/5/22
to SDFABRIC-Dev, d.l...@surrey.ac.uk, charle...@intel.com
Hi David,


Thanks for your debugging process and I believe you hit the following issue when you use a more high k8s version than what we used.

Kubernetes change its default cgroup manager from cgroup to system since v1.22, however docker still uses the cgroup as its cgroup manager, and it's not recommended to run two different cgroup manager at the same time.
The solution is what you used, re-configure the docker to force it to use the system manager.
For the Kubernetes 1.24, the docker isn't supported by default, the user should choose another CRI runtime like containerd or CRI-O, both you may have to configure and install by yourself in the ONL.

For now, I think we should mention the range of the Kubernetes version like 1.18 to 1.21 in our document, and then upgrade the ONL to make it support the higher k8s version in the future.

The document was generated automatically and you can check its source here

If you're interested in contributing to our document, please let me know,  I can point you to where to start and how to contribute.

Thanks
HungWei 






On Friday, July 1, 2022 at 4:26:08 AM UTC-7 d.l...@surrey.ac.uk wrote:

Success!

 

I’ve finally been able to install Kubernetes on the ONL system.  The latest K8S is 1.24.2 but this will not run on Debian Stretch with Docker 20.x.x which is the version ONL is based on so I went back to Docker 19.03.15 and K8S 1.18.20.

 

As my CP node is Ubuntu 20.04, I’ve had to force installation of older packages:

 

                apt-get install containerd=1.3.3-0ubuntu2

apt-get install docker.io=19.03.8-0ubuntu1

 

On both the switch and the CP node I installed:

 

                apt install kubectl=1.18.20-00 kubelet=1.18.20-00 kubernetes-cni cri-tools=1.19.0-00 kubeadm=1.18.20-00

 

I also found that I needed to change the cgroups driver to systemd for Docker on the CP node in /etc/docker/daemon.json:

{

  "exec-opts": ["native.cgroupdriver=systemd"]

}

 

I’m using Flannel networking but found that the initnetdev script was missing in ONL – I added it from here https://github.com/opennetworklinux/ONL/blob/master/builds/swi/all/shared/sbin/initnetdev

 

Once that was installed, I ran
                     kubeadm init --pod-network-cidr 10.244.0.0/16

.. on the Control Plane node and the corresponding kubeadm on the switch.  I then installed Flannel:

 

                kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

 

I’m now in the process of labelling and tainting the switch before I attempt to deploy SD-Fabric.

 

I hope this helps anyone else going through the same issues.   If ONL is bumped to Debian 10 at any point, then it would be possible to run K8S 1.24 and newer Docker so these steps will change.

 

David

 


Date: Thursday, 30 June 2022 at 06:39

To unsubscribe from this group and stop receiving emails from it, send an email to sdfabric-announce+unsubscribe@opennetworking.org.


To view this discussion on the web visit

--
You received this message because you are subscribed to the Google Groups "SDFABRIC-Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sdfabric-dev+unsubscribe@opennetworking.org.

--
You received this message because you are subscribed to the Google Groups "SDFABRIC-Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sdfabric-dev+unsubscribe@opennetworking.org.

--
You received this message because you are subscribed to the Google Groups "SDFABRIC-Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sdfabric-dev+unsubscribe@opennetworking.org.

David Lake

unread,
Jul 6, 2022, 3:31:15 AM7/6/22
to Hung-Wei Chiu, SDFABRIC-Dev, charle...@intel.com

Hi HungWei

 

I would be very happy to contribute to the documentation.

 

My Kubernetes Control Plane node is Ubuntu 22.04 which by default installs Kubernetes 1.24.2.  I was able to install 1.24.2 on the ONL with Debian 9 but I didn’t know about the Docker issue at that point and reverted to Kubernetes 1.18.8 across both the control plane and ONL.

 

I have had to put ‘apt-mark hold’ for  kubeadm, kubectl, Kubernetes-cri, kubelet and docker.io to prevent them upgrading automatically.

 

Is there any way that Kubernetes plus docker with a systemd configuration could be included in the base ONL package rather than having to install?

 

Thanks

 

David

To: Lake, David (PG/R - Elec Electronic Eng) <d.l...@surrey.ac.uk>, sdfabr...@opennetworking.org <sdfabr...@opennetworking.org>

To unsubscribe from this group and stop receiving emails from it, send an email to sdfabric-annou...@opennetworking.org.

--
You received this message because you are subscribed to the Google Groups "SDFABRIC-Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sdfabric-dev...@opennetworking.org.

--
You received this message because you are subscribed to the Google Groups "SDFABRIC-Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sdfabric-dev...@opennetworking.org.

--
You received this message because you are subscribed to the Google Groups "SDFABRIC-Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sdfabric-dev...@opennetworking.org.

Chan, Charles

unread,
Jul 7, 2022, 12:12:06 AM7/7/22
to David Lake, Hung-Wei Chiu, SDFABRIC-Dev

Hi David,

We don’t plan to upgrade to a newer ONL image. Instead, we are looking into porting Stratum onto SONiC. That will provide even better platform support moving forward.
I would suggest sticking with current ONL + old K8s for a while, or you can try to build Stratum on top of Ubuntu 22.04 – but we won’t be able to support that either.

Thanks.

 

Intel

Charles Chan, PhD (he/him)

Cloud Software Development Engineer

Software Product Incubation | Network and Edge Group

Intel Corporation  |  intel.com

 

Reply all
Reply to author
Forward
0 new messages