deployment of oran powder profile
Robin Wiebusch
Dear powder team and members,
I am currently working on solutions for deploying O-RAN (https://docs.o-ran-sc.org/en/latest/) based RICs in combination with srslte (https://www.srsran.com/). My ultimate goal is to develop xapps for 5G Network Slicing for research purposes. In this context, I came across the O-RAN powder profile, which I am trying to run on a local virtual machine for testing purposes: https://www.powderwireless.net/show-profile.php?project=PowderProfiles&profile=O-RAN.
Yet after thoroughly following the explanations, I could not get the nexran slicing xapp to work properly.
I would be thankful to get information on dependencies which I might have missed as well as on recommended versions and known bug fixes.
System:
Ubuntu 20.0.4
helm version 3.7.2
kubectl:
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
I used the following software according to the listed software of the profile:
srslte:
powder srslte fork: https://gitlab.flux.utah.edu/powderrenewpublic/srslte-ric
I utilized the given e2 bindings as well as the asn1c compiler: https://gitlab.eurecom.fr/oai/asn1c.git, commit f12568d617dbf48497588f8e227d70388fa217c9
As suggested, I use the zmq virtual radios https://docs.srsran.com/en/latest/app_notes/source/zeromq/source/index.html
During make test, I got the following error message of which I am not certain whether it is critical:
The following tests FAILED: 771 - nas_test (SEGFAULT) Errors while running CTest
O-RAN RIC platform:
for RIC deployment, I used the following fork: https://gitlab.flux.utah.edu/powderrenewpublic/dep
here, I used the cherry-powder release, because in dawn-powder release I was not able to get the e2term pod to work.
after instantiating the RIC platform and running both srsepc and srsenb, the logs of the e2term pod show the error
{"ts":1644939386846,"crit":"ERROR","id":"E2Terminator","mdc":{"thread id":"140040957060864"},"msg":"epoll error, events 8 on fd 20, RAN NAME : enB_macro_001_001_0019b0"}
nexran xapp:
I use the following xapp: https://gitlab.flux.utah.edu/powderrenewpublic/nexran
When following the profile tutorial, I am able to call the nexran api and configure the slices, yet the UE’s bandwidth does not seem to change depending on the slice priorities.
Is there anything crucial that I might miss or are there other software components required?
Are there any suggestions on the occuring errors?
I am very thankful for any suggestions.
Kind regards,
Robin
David M. Johnson
> Dear powder team and members,
>
> I am currently working on solutions for deploying O-RAN
> <https://docs.o-ran-sc.org/en/latest/>) based RICs in combination with
> srslte (<https://www.srsran.com/>https://www.srsran.com/
> <https://www.srsran.com/>). My ultimate goal is to develop xapps for 5G
> the O-RAN powder profile, which I am trying to run on a local virtual
> machine for testing purposes:
> <https://www.powderwireless.net/show-profile.php?project=PowderProfiles&profile=O-RAN>.
> Yet after thoroughly following the explanations, I could not get the
> nexran slicing xapp to work properly.
>
> I would be thankful to get information on dependencies which I might
> have missed as well as on recommended versions and known bug fixes.
tried the profile, to make sure you can at least get it working there?
> _System_:
> Ubuntu 20.0.4
>
> helm version 3.7.2
>
> kubectl:
>
> GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77",
> GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z",
> GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
>
> GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77",
> GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z",
> GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
>
>
> I used the following software according to the listed software of the
> profile:
>
>
> powder srslte fork:
> <https://gitlab.flux.utah.edu/powderrenewpublic/srslte-ric>https://gitlab.flux.utah.edu/powderrenewpublic/srslte-ric
>
> I utilized the given e2 bindings as well as the asn1c compiler:
> <https://gitlab.eurecom.fr/oai/asn1c.git>, commit
>
> As suggested, I use the zmq virtual radios
> <https://docs.srsran.com/en/latest/app_notes/source/zeromq/source/index.html>
>
> During /make test/, I got the following error message of which I am not
>
> running CTest/
>
> _O-RAN RIC platform_:
> for RIC deployment, I used the following fork:
>
> here, I used the cherry-powder release, because in dawn-powder release I
> was not able to get the e2term pod to work.
>
> after instantiating the RIC platform and running both srsepc and srsenb,
> the logs of the e2term pod show the error
>
This is likely the problem; probably the E2 agent in srsenb is losing
its SCTP connection within the first few seconds. You should see
confirmation in the srsenb's logs if you have the right debug stuff set.
This should not happen in some network plugins, e.g. calico, but does
happen in others like flannel. You can workaround this by forcing
srsenb to bind the SCTP socket to a local addr and port with additional
args to srsenb: `--ric.agent.local_ipv4_addr <addr>
--ric.agent.local_port <port>`.
> _nexran xapp:_
> I use the following xapp:
>
> When following the profile tutorial, I am able to call the nexran api
> and configure the slices, yet the UE’s bandwidth does not seem to change
> depending on the slice priorities.
and control messages from the xApp won't make it to the srsenb.
I would advise that you first create the NodeB object and make sure your
xApp is receiving KPM indications every five seconds, to verify that
your E2 channel is stable.
> Is there anything crucial that I might miss or are there other software
> components required?
in many Kubernetes configurations -- only the calico and flannel network
plugins with multus enabled, metallb, etc.
> Are there any suggestions on the occuring errors?
>
> I am very thankful for any suggestions.
>
> Kind regards,
>
> Robin
Robin Wiebusch
Hi David,
thank you very much for the detailed reply. In the following I refer to your suggestion:
> “This is likely the problem; probably the E2 agent in srsenb is losing
> its SCTP connection within the first few seconds. You should see
> confirmation in the srsenb's logs if you have the right debug stuff set.
> This should not happen in some network plugins, e.g. calico, but does
> happen in others like flannel. You can workaround this by forcing
> srsenb to bind the SCTP socket to a local addr and port with additional
> args to srsenb: `--ric.agent.local_ipv4_addr <addr> -ric.agent.local_port <port>`.”
Indeed I was able to confirm that the SCTP connection is lost using the logs you suggested.
In order to solve it, I switched from flannel to calico, which took me some time to figure out. In fact, I needed to use some older software versions in order to get it to work properly (Ubuntu 18, Kubernetes 1.16. and Calico 3.16).
Unfortunately, this did not solve the problem. I attached detailed error logs; I would be very thankful if you could have another look at it and give me further advice.
I also tried your second suggestion of forcing to bind the SCTP socket to a local addr and port. However, the following error occurs additionally to the error logs mentioned above:
“Failed to bind on address 192.168.128.1: Cannot assign requested address errno 99”
I do not entirely understand whether I need to use a specific address and port here, could you please explain?
Thank you in advance!
Best regards,
Robin
David M. Johnson
> Hi David,
>
> thank you very much for the detailed reply. In the following I refer to
> your suggestion:
>
>
>> “This is likely the problem; probably the E2 agent in srsenb is losing
>> its SCTP connection within the first few seconds. You should see
>> confirmation in the srsenb's logs if you have the right debug stuff set.
>> This should not happen in some network plugins, e.g. calico, but does
>> happen in others like flannel. You can workaround this by forcing
>> srsenb to bind the SCTP socket to a local addr and port with additional
>> args to srsenb: `--ric.agent.local_ipv4_addr <addr>
> -ric.agent.local_port <port>`.”
>
> Indeed I was able to confirm that the SCTP connection is lost using the
> logs you suggested.
>
> In order to solve it, I switched from flannel to calico, which took me
> some time to figure out. In fact, I needed to use some older software
> versions in order to get it to work properly (Ubuntu 18, Kubernetes
> 1.16. and Calico 3.16).
>
> Unfortunately, this did not solve the problem. I attached detailed error
> logs; I would be very thankful if you could have another look at it and
> give me further advice.
initial connection works fine. Maybe start up a POWDER experiment and
(broadly) compare its k8s config against yours? I know things work in
calico/flannel the way we configure them, but I don't know all the
stumbling blocks here with SCTP. At a high level, I can say our default
config is to use ipvs as the kube proxy mode; to enable multus; enable
dockerd iptables; and to enable metallb to provide public IPs (and to
tweak ARP settings to make metallb work). And my overall thinking is
that kube-proxy is failing to make the connection persist into the pod
-- e.g. a packet in the NAT path is getting dropped instead of
translated, maybe not being recognized as part of an existing connection
-- and this winds up being experienced by srsenb as a connection close
(SCTP ABORT). The mechanism down in the kernel that is maintain this
"existing connection" state is called `conntrack`. You could check to
see that SCTP conntrack support is built into your kernel
(CONFIG_NF_CT_PROTO_SCTP=y) (and if using kube-proxy ipvs mode,
CONFIG_IP_VS_PROTO_SCTP=y). But these would already be enabled on any
distro kernel, so unlikely to be a problem unless you are building a
custom kernel.
> I also tried your second suggestion of forcing to bind the SCTP socket
> to a local addr and port. However, the following error occurs
> additionally to the error logs mentioned above:
>
> “Failed to bind on address 192.168.128.1: Cannot assign requested
> address errno 99”
you tried to bind to a nonexistent local interface or address. So maybe
try again and verify that the addr is correct?
> I do not entirely understand whether I need to use a specific address
> and port here, could you please explain?
https://wiki.o-ran-sc.org/display/GS/Near+Realtime+RIC+Installation?focusedCommentId=47746313#comment-47746313
. I don't know the details of flannel so I can only speculate past that
comment. (Pratheek provide a patch to our srslte fork that added the
local ip/port bind args necessary to solve this case for flannel.)
> Robin
David
> You received this message because you are subscribed to the Google
> Groups "Powder Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to powder-users...@googlegroups.com
> <mailto:powder-users...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/powder-users/816cc418-e85b-4ef6-aaf0-57d0c5e728e2n%40googlegroups.com
> <https://groups.google.com/d/msgid/powder-users/816cc418-e85b-4ef6-aaf0-57d0c5e728e2n%40googlegroups.com?utm_medium=email&utm_source=footer>.
Robin Wiebusch
Dear David,
thanks again for the great help. As already discussed in your office hour I was able to fix the sctp error. This is a summary of what I did to achieve it:
Software versions:
ubuntu 18
kubernetes v1.16.15
flannel or calico v3.16
e2term image provided by powder (5.4.8-powder), which is the cherry release
The dawn release version (5.4.9-powder) did not work due to some rmr errors occuring on pod startup (1649878294366 22/RMR [INFO] ric message routing library on SI95 p=38000 mv=3 flg=00 id=a (11838bc 4.7.4 built: Apr 27 2021) terminate called without an active exception)
First I enabled iptables, multus and metallb (and configured it as explained here https://metallb.universe.tf/configuration/)
Then, when starting the srsenb, I passed the ip and port of the interface used by the E2 agent using the parameters --ric.agent.local_ipv4_addr=<ip> and --ric.agent.local_port=59596
I tried it with both flannel and calico, and in both cases I had to pass these params, otherwise the aforementioned sctp error would occur (19:05:42.314871 [RIC ] [E] Error reading from SCTP socket: Connection reset by peer). In the flannel case it is the “cni0” interface, and in the calico case it is the “vxlan.calico” interface.
As we’ve talked about, another error occurs when trying to use the nexran xapp. I attached the relevant log files. Restarting some parts of the application did not solve it, unfortunately (I tried restarting / redeploying the ric, the xapp, and both multiple times). Here are some observations which might help to identify the problem in hope that you might have another idea on how to fix it:
- The error shown in the e2term logs occurs immediately after I create the srsenb in the nexran xapp using curl -i -X POST -H "Content-type: application/json" -d '{"type":"eNB","id":411,"mcc":"001","mnc":"01"}' <http://$>{NEXRAN_XAPP}:8000/v1/nodebs ; echo ; echo
- warnings/errors during building of srslte:
- after the cmake command: warning “Manually-specified variables were not used by the project: RIC_GENERATED_E2SM_NI_BINDING_DIR”
- During make test: “The following tests FAILED: 771 - nas_test (SEGFAULT) Errors while running CTest”. In your nexran paper, you explained that the nodeb is modified to decode NAS messages to capture the UE IMSIs. Maybe this could be related somehow?
- The influxdb pod stays in pending status. I was not able to fix this yet.
Additionally, here are some potential errors or deviations from what worked for me which I found in the oran profile description (https://www.powderwireless.net/show-profile.php?project=PowderProfiles&profile=O-RAN). I’d like to share these to help improve the description.
- In order to talk to the nexran xapp via the rest api, it is suggested to get the ip of the service-ricxapp-nexran-rmr service. However, I achieved it using the service-ricxapp-nexran-nbi service: export NEXRAN_XAPP=kubectl get svc -n ricxapp --field-selector metadata.name=service-ricxapp-nexran-nbi -o jsonpath='{.items[0].spec.clusterIP}' ; echo $NEXRAN_XAPP
- it is mentioned that the slice share can be set between 1 - 1024, however 0 - 1024 seems to be the available range.
Again, I am very thankful for any ideas or suggestions!
Best regards,
Robin