Issues running SD-Fabric

46 views
Skip to first unread message

David Lake

unread,
Aug 15, 2022, 7:27:56 AM8/15/22
to sdfabr...@opennetworking.org

I’m trying to run SD-Fabric as a UP4 with two EdgeCore switches and a single worker running ONOS.

 

My k8s nodes and taints are:

 

root@cp-node:/home/admin5g/sdfabric-helm-charts/sdfabric# kubectl get node -o custom-columns=NAME:.metadata.name,TAINT:.spec.taints

NAME      TAINT

5g1       [map[effect:NoSchedule key:node-role.kubernetes.io value:switch]]

5g2       [map[effect:NoSchedule key:node-role.kubernetes.io value:switch]]

cp-node   [map[effect:NoSchedule key:node-role.kubernetes.io/master]]

onos      <none>

root@cp-node:/home/admin5g/sdfabric-helm-charts/sdfabric# kubectl get nodes

NAME      STATUS   ROLES    AGE   VERSION

5g1       Ready    worker   45d   v1.18.20

5g2       Ready    worker   42d   v1.18.20

cp-node   Ready    master   45d   v1.18.20

onos      Ready    worker   44d   v1.18.20

root@cp-node:/home/admin5g/sdfabric-helm-charts/sdfabric#

 

 

My yaml configuration file is attached.

 

When I try to load the system, I get a ‘CrashLoopBackoff’ from the PFCP agent and because I never see ‘readiness’ from it, none of the other configuration is implemented:

 

root@cp-node:/home/admin5g/sdfabric-helm-charts/sdfabric# kubectl get pods -n sdfabric

NAME                                                        READY   STATUS             RESTARTS   AGE

sdfabric-atomix-0                                           1/1     Running            0          4m7s

sdfabric-onos-classic-0                                     1/1     Running            0          4m7s

sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f   1/1     Running            0          4m7s

sdfabric-pfcp-agent-0                                       0/1     CrashLoopBackOff   5          4m7s

 

This is what I see in the log:

 

m26s       Normal    SuccessfulCreate    statefulset/sdfabric-atomix                                      create Pod sdfabric-atomix-0 in StatefulSet sdfabric-atomix successful

2m26s       Normal    Scheduled           pod/sdfabric-pfcp-agent-0                                        Successfully assigned sdfabric/sdfabric-pfcp-agent-0 to onos

2m26s       Normal    SuccessfulCreate    statefulset/sdfabric-onos-classic                                create Pod sdfabric-onos-classic-0 in StatefulSet sdfabric-onos-classic successful

2m26s       Normal    ScalingReplicaSet   deployment/sdfabric-onos-classic-onos-config-loader              Scaled up replica set sdfabric-onos-classic-onos-config-loader-59d64bd8dc to 1

2m26s       Normal    SuccessfulCreate    replicaset/sdfabric-onos-classic-onos-config-loader-59d64bd8dc   Created pod: sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f

2m26s       Normal    Scheduled           pod/sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f    Successfully assigned sdfabric/sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f to onos

2m26s       Normal    Scheduled           pod/sdfabric-atomix-0                                            Successfully assigned sdfabric/sdfabric-atomix-0 to onos

2m26s       Normal    Scheduled           pod/sdfabric-onos-classic-0                                      Successfully assigned sdfabric/sdfabric-onos-classic-0 to onos

2m26s       Normal    SuccessfulCreate    statefulset/sdfabric-pfcp-agent                                  create Pod sdfabric-pfcp-agent-0 in StatefulSet sdfabric-pfcp-agent successful

2m25s       Normal    Started             pod/sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f    Started container onos-config-loader

2m25s       Warning   FailedMount         pod/sdfabric-onos-classic-0                                      MountVolume.SetUp failed for volume "probe-scripts" : failed to sync configmap cache: timed out waiting for the condition

2m25s       Warning   FailedMount         pod/sdfabric-onos-classic-0                                      MountVolume.SetUp failed for volume "init-scripts" : failed to sync configmap cache: timed out waiting for the condition

2m25s       Warning   FailedMount         pod/sdfabric-onos-classic-0                                      MountVolume.SetUp failed for volume "logging-configmap" : failed to sync configmap cache: timed out waiting for the condition

2m25s       Normal    Created             pod/sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f    Created container onos-config-loader

2m25s       Normal    Pulled              pod/sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f    Container image "opencord/onos-classic-helm-utils:0.1.0" already present on machine

2m24s       Normal    Pulled              pod/sdfabric-atomix-0                                            Container image "atomix/atomix:3.1.12" already present on machine

2m24s       Normal    Pulled              pod/sdfabric-atomix-0                                            Container image "ubuntu:16.04" already present on machine

2m24s       Normal    Created             pod/sdfabric-atomix-0                                            Created container configure

2m24s       Normal    Created             pod/sdfabric-atomix-0                                            Created container atomix

2m24s       Normal    Started             pod/sdfabric-atomix-0                                            Started container atomix

2m24s       Normal    Started             pod/sdfabric-atomix-0                                            Started container configure

2m23s       Normal    Started             pod/sdfabric-onos-classic-0                                      Started container onos-classic-init

2m23s       Normal    Created             pod/sdfabric-onos-classic-0                                      Created container onos-classic-init

2m23s       Normal    Pulled              pod/sdfabric-onos-classic-0                                      Container image "tutum/dnsutils:latest" already present on machine

2m22s       Normal    Started             pod/sdfabric-onos-classic-0                                      Started container onos-classic-logging

2m22s       Normal    Created             pod/sdfabric-onos-classic-0                                      Created container onos-classic-logging

2m22s       Normal    Pulled              pod/sdfabric-onos-classic-0                                      Container image "alpine" already present on machine

2m21s       Normal    Started             pod/sdfabric-onos-classic-0                                      Started container onos-classic

2m21s       Normal    Pulled              pod/sdfabric-onos-classic-0                                      Container image "opennetworking/sdfabric-onos:1.1.1" already present on machine

2m21s       Normal    Created             pod/sdfabric-onos-classic-0                                      Created container onos-classic

109s        Warning   Unhealthy           pod/sdfabric-onos-classic-0                                      Readiness probe failed: {"id":"onos-0","ip":"10.244.2.34","tcpPort":9876,"status":"READY","lastUpdate":"1660562485597","humanReadableLastUpdate":"6s ago"}

{"message":"ready"}

{"message":"ready"}

{"message":"ready"}

{"message":"ready"}

{"message":"ready"}

{"message":"pending"}

org.onosproject.segmentrouting is not yet ready

94s         Warning   Unhealthy           pod/sdfabric-onos-classic-0                                      Readiness probe failed: {"id":"onos-0","ip":"10.244.2.34","tcpPort":9876,"status":"READY","lastUpdate":"1660562485597","humanReadableLastUpdate":"20s ago"}

{"message":"ready"}

{"message":"ready"}

{"message":"ready"}

{"message":"ready"}

{"message":"ready"}

{"message":"ready"}

{"message":"ready"}

{"message":"pending"}

org.stratumproject.fabric-tna is not yet ready

90s         Warning   Unhealthy           pod/sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f    Readiness probe failed:

54s         Normal    Pulled              pod/sdfabric-pfcp-agent-0                                        Container image "omecproject/upf-epc-pfcpiface:master-2f04e3a" already present on machine

54s         Normal    Created             pod/sdfabric-pfcp-agent-0                                        Created container sdfabric-pfcp-agent

54s         Normal    Started             pod/sdfabric-pfcp-agent-0                                        Started container sdfabric-pfcp-agent

41s         Warning   BackOff             pod/sdfabric-pfcp-agent-0                                        Back-off restarting failed container

21s         Normal    NoPods              poddisruptionbudget/sdfabric-atomix-pdb                          No matching pods found

 

Not sure what I’m doing wrong and would appreciate some help?

 

Thank you

 

David

 

 

signature_3900680105

5G & 6G Innovation Centres

Institute for Communication Systems (ICS)
University of Surrey
Guildford
GU2 7XH

 

myvalues.yaml.rtf

David Lake

unread,
Aug 15, 2022, 10:56:26 AM8/15/22
to sdfabr...@opennetworking.org

I think I’ve found the cause of the pfcp-agent crash but I’ve no idea how to fix it.

 

root@cp-node:/home/admin5g/sdfabric-helm-charts/sdfabric# kubectl logs sdfabric-pfcp-agent-0 -p -n sdfabric

unable to retrieve container logs for docker://86f5fc74c403b16970b639e56b4be03d8c308d133ab2cc8b081cb29314895bfbroot@cp-node:/home/admin5g/sdfabric-helm-charts/sdfabric# kubectl logs sdfabric-pfcp-agent-0 -p -n sdfabric

time="2022-08-15T14:00:01Z" level=info msg="{Mode: AccessIface:{IfName:} CoreIface:{IfName:} CPIface:{Peers:[] UseFQDN:false NodeID:pfcp-agent HTTPPort:8080 Dnn:internet EnableUeIPAlloc:false UEIPPool:10.250.0.0/24} P4rtcIface:{SliceID:0 AccessIP:172.17.0.1/32 P4rtcServer:sdfabric-onos-classic-hs P4rtcPort:51001 QFIToTC:map[] DefaultTC:3} EnableP4rt:true EnableFlowMeasure:false SimInfo:{MaxSessions:0 StartUEIP:<nil> StartENBIP:<nil> StartAUPFIP:<nil> N6AppIP:<nil> N9AppIP:<nil> StartN3TEID: StartN9TEID:} ConnTimeout:0 ReadTimeout:15 EnableNotifyBess:false EnableEndMarker:false NotifySockAddr: EndMarkerSockAddr: LogLevel:info QciQosConfig:[] SliceMeterConfig:{N6RateBps:0 N6BurstBytes:0 N3RateBps:0 N3BurstBytes:0} MaxReqRetries:5 RespTimeout:2s EnableHBTimer:false HeartBeatInterval:}" func=main.main file="/pfcpiface/cmd/pfcpiface/main.go:37"

time="2022-08-15T14:00:01Z" level=fatal msg="Unable to resolve hostname pfcp-agent lookup pfcp-agent on 10.96.0.10:53: no such host" func=github.com/omec-project/upf-epc/pfcpiface.NewUPF file="/pfcpiface/pfcpiface/upf.go:109"

 

Can someone advise on this please?

 

Thanks


David

David Lake

unread,
Aug 15, 2022, 1:04:58 PM8/15/22
to sdfabr...@opennetworking.org

And another problem.

 

ONOS is running – I can ssh into it.

David

 

 

kubectl logs --namespace=sdfabric sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f

 

Waiting for ONOS Cluster to be ready

-----------------------------------------

Loading netcfg into ONOS

-----------------------------------------

cat: can't open '/opt/configs/netcfg.json': No such file or directory

sed: can't read /opt/configs/netcfg.json: No such file or directory

Update is unnecessary since the netcfg on the ONOS cluster is up to date

-----------------------------------------

Loading component configs into ONOS

-----------------------------------------

-----------------------------------------

Check that component netcfg is active

-----------------------------------------

curl: (22) The requested URL returned error: 404

Waiting for netcfg to be active

curl: (22) The requested URL returned error: 404

Waiting for netcfg to be active

curl: (22) The requested URL returned error: 404

Waiting for netcfg to be active

 

 

This just repeats

--
You received this message because you are subscribed to the Google Groups "SDFABRIC-Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sdfabric-dev...@opennetworking.org.
To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/sdfabric-dev/DB7PR06MB4792F9DAB4B8FC5D9174D9B5B5689%40DB7PR06MB4792.eurprd06.prod.outlook.com.

Reply all
Reply to author
Forward
0 new messages