[stratum-dev] Question about building stratum standalone

[A Sydney]

Hi Stratum folks,

                          I'm attempting to use the stratum standalone instance (https://github.com/stratum/stratum/blob/main/stratum/hal/bin/bmv2/README.md) to setup a topology on a baremetal server. Can you kindly provide feedback to the following:

1. Where can I find "p4_pipeline.pb.txt" to run the binary? In particular, as a simple test, I'm attempting to use the main.p4 file shipped with the ngsdn-tutorial (https://github.com/opennetworkinglab/ngsdn-tutorial). On the tutorial VM, generated the corresponding bmv2.json and p4info.txt (i.e. using make p4-build). Was there some other "p4_pipeline.pb.txt" file generated that I can use on the bare metal setup? Or perhaps, I have to run bmv2.json through some postprocess to get "p4_pipeline.pb.txt"?

2. Assuming I've got a running binary with all the arguments sorted, how can I go about creating a topology of switches? Can I simply create containers/VMs (i.e. with an linux OS), clone the stratum repo on each device, add container/VM interfaces in "chassis_config.pb.txt", and run the bmv2 executable from each device?

Thanks!

-Syd

[Maximilian Pudelko]

Hi Syd,

> 1. Where can I find "p4_pipeline.pb.txt" to run the binary?

That file is the place where Stratum will store a pushed pipeline. It can only be set via the P4Runtime SetForwardingPipelineConfig RPC call.

The Running the stratum_bmv2 binary section of the README gives an example:

# run P4Runtime client
cp stratum/hal/bin/bmv2/update_config.py /tmp/ && \
[sudo] docker run -v /tmp:/tmp -w /tmp p4lang/pi ./update_config.py \
    --grpc-addr <YOUR_HOST_IP_ADDRESS>:9559 --json <prog>.json --p4info <prog>.proto.txt

You could also use the p4runtime-shell.

> 2 .
Yes, the bmv2 binary should bind to all interfaces given in the chassis_config.pb.txt.
With some wiring it should be possible to build a web of software switches.
But instead of building Stratum many times, check if the Debian package might not be more convenient:
https://github.com/stratum/stratum/blob/main/stratum/hal/bin/bmv2/BUILD#L81-L98

There is also a Stratum-enabled Mininet version, which takes care of a lot of these setup tasks. The

NGSDN tutorial you linked uses it, too.

Best,

Max

_______________________________________________
stratum-dev mailing list
strat...@lists.stratumproject.org
https://lists.stratumproject.org/listinfo/stratum-dev

[A Sydney]

I'm still having a few issues:

1. Please take note of the following: 

 a) On my host, I've created ~/stratum/bmv2_config (i.e. in the top level stratum directory) where I've added bmv2.json and p4info.txt for loopback.p4. I've also added config.txt which contains the ports I would like stratum to use.

 b) In the stratum container (/stratum), I run the following:

stratum> sudo ./bazel-bin/stratum/hal/bin/bmv2/stratum_bmv2 \
    --persistent_config_dir=/stratum/bmv2_config \  
    --chassis_config_file=/stratum/bmv2_config/config.txt  \
    --bmv2_log_level=debug

And see the following expected error (i.e. "expected", since we will generate this file subsequently):

"""

/var/run/stratum/pipeline_cfg.pb.txt not found..

"""

c) Now back on my host system, I do the following:

'"""

host> cp stratum/hal/bin/bmv2/update_config.py /tmp/ && \

docker run -v /tmp:/tmp -w /tmp p4lang/pi ./update_config.py \

--grpc-addr 52.117.10.174:9559 --json bmv2_config/bmv2.json --p4info bmv2_config/p4info.txt

"""

And get the following error:

"""

Traceback (most recent call last):
  File "./update_config.py", line 10, in <module>
    from p4.v1 import p4runtime_pb2
ImportError: No module named p4.v1
"""

I Googled this error and looks like folks have come across this (https://github.com/p4lang/tutorials/issues/178) and it appears that perhaps P4Runtime is not installed properly? 

Questions:

* Is my workflow for starting the bmv2 switch and generating pipeline_cfg.pb.txt correct?

* Should I hop onto the container and run the install? Or is there an easier way around this issue?

* If "c" was correct, then "pipeline_cfg.pb.txt" would automatically be generated and inserted in the switch. Would I also have to perform c) for all switches?

Thanks,

-Syd

[Maximilian Pudelko]

> Is my workflow for starting the bmv2 switch and generating pipeline_cfg.pb.txt correct?

looks ok to me. You might still want to set the --forwarding_pipeline_configs_file to somewhere

writable by Stratum.

> Should I hop onto the container and run the install? Or is there an easier way around this issue?

The update_config.py scripts are pretty old, unmaintained and broken as you just found out. I'd recommend

using the P4Runtime-shell instead.

> If "c" was correct, then "pipeline_cfg.pb.txt" would automatically be generated and inserted in the switch. Would I also have to perform c) for all switches?

Yes, each switch needs to have a pipeline.

Max

[A Sydney]

So I grabbed the P4R shell from ngsdn-tutorial, and I've got progress. However, I can't connect to the grpc server on the switch. In particular, when I fire-up the switch in the container, I see the following:

"""

E20210322 21:51:44.785127   762 hal.cc:221] Stratum external facing services are listening to 0.0.0.0:28000, 0.0.0.0:9339, 0.0.0.0:9559, localhost:28000...

"""

I then attempt to connect to from my host system and get the following error:

"""

CRITICAL:root:Failed to establish session with server
CRITICAL:root:StreamChannel error, closing stream
CRITICAL:root:P4Runtime RPC error (UNAVAILABLE): failed to connect to all addresses

"""

which makes a bit of sense since I can't see any of the stratum external ports from my host (i.e. for example "netstat -an | grep 9339" has no results). Thoughts?

Thanks!

-Syd

[A Sydney]

Update:

Summary: I used the "-p" argument for "docker run" to connect a host port to that of a port on the container, and everything works so far.  Below are a few details for other folks who may have this same issue:

* As a quick test, I updated the last line of ~/stratum/setup_dev_env.sh as follows to use port 50001 of the host and docker container:

host> vim setup_dev_env.sh

...

docker run $DOCKER_RUN_OPTIONS  -p 0.0.0.0:50001:50001/tcp --cap-add=NET_ADMIN -w /stratum --user $USER -ti $IMAGE_NAME bash

* After setting up the container, I run the switch as follows: 

stratum> ./bazel-bin/stratum/hal/bin/bmv2/stratum_bmv2 \
    --persistent_config_dir=/stratum/bmv2_config \
    --forwarding_pipeline_configs_file=/stratum/bmv2_config/p4_pipeline_leaf1.pb.txt \
    --chassis_config_file=/stratum/bmv2_config/config.txt \
    --external_stratum_urls="0.0.0.0:50001" \
    --bmv2_log_level=debug

* From my host, I can now see port 50001 available (i.e. using "netstat -an | grep 50001"). I can also connect to the switch as follows:

host> cd ~/ngsdn-tutorial

host> util/p4rt-sh  --grpc-addr 0.0.0.0:50001 --config ../stratum/bmv2_config/p4info.txt,../stratum/bmv2_config/bmv2.json --election-id 0,1

Note: I'm running P4Runtime from the ngsdn-tutorial repo.

PS. If there is some other elegant way of doing this, please provide feedback.

Cheers!

-Syd

On Wed, Mar 24, 2021 at 2:19 PM A Sydney via stratum-dev <strat...@lists.stratumproject.org> wrote:

So I grabbed the P4R shell from ngsdn-tutorial, and I've got progress. However, I can't connect to the grpc server on the switch. In particular, when I fire-up the switch in the container, I see the following:

"""

E20210322 21:51:44.785127   762 hal.cc:221] Stratum external facing services are listening to 0.0.0.0:28000, 0.0.0.0:9339, 0.0.0.0:9559, localhost:28000...

"""

I then attempt to connect to from my host system and get the following error:

"""

CRITICAL:root:Failed to establish session with server
CRITICAL:root:StreamChannel error, closing stream
CRITICAL:root:P4Runtime RPC error (UNAVAILABLE): failed to connect to all addresses

"""

which makes a bit of sense since I can't see any of the stratum external ports from my host (i.e. for example "netstat -an | grep 9339" has no results). Thoughts?

Thanks!

-Syd

[A Sydney]

But instead of building Stratum many times, check if the Debian package might not be more convenient:
https://github.com/stratum/stratum/blob/main/stratum/hal/bin/bmv2/BUILD#L81-L98

I didn't quite get this one. Where can I get the associated *.deb package for the install?

Thanks,

-Syd

[Maximilian Pudelko]

Hi Syd,

As you have found out, the P4Runtime/gNMI ports have to be exposed from inside the container

to be reachable from outside.

We have a start-stratum-container.sh script that takes care of that, or you map them manually with -p,

like you did.

> I didn't quite get this one. Where can I get the associated *.deb package for the install?

What I linked is the Bazel target that builds the .deb package. Build it with

`bazel build --stamp //stratum/hal/bin/bmv2:stratum_bmv2_debian`.

The resulting file will be under `bazel-bin/stratum/hal/bin/bmv2/stratum_bmv2_debian.deb`

and can be installed on most Ubuntu and Debian systems:

`apt-get install --reinstall ./stratum_bmv2_debian.deb`

Best,

Max

[A Sydney]

Ah great! I was able to build the package, yay! But I'm having a few dependency issues:

On the vm which hosts the stratum switch, I'm running Debian 10. Unfortunately, I could not find "libboost1.55-all-dev. However, libboost1.67-all-dev did exist, so I attempted to install this version but no success. It appears that "libboost1.55-all-dev" is a hard requirement?

"""

sudo apt-get install --reinstall ./stratum_bmv2_debian.deb

...

Note, selecting 'stratum_bmv2' instead of './stratum_bmv2_debian.deb'

...

stratum_bmv2 : Depends: libboost1.55-all-dev but it is not installable

...

"""

Any suggestions on how to work around this one? If this version is truly a hard requirement, what linux distributions would you suggest (e.g. what distributions have you tried the install)?

Thanks,

-Syd

[Maximilian Pudelko]

We're using Debian 9 in many parts of the stack. You could either use that or

replace the dependency with `libboost1.67-all-dev`in the Bazel build rule.

Max

[A Sydney]

Update:

I was able to install the Stratum BMv2 switch as a standalone instance on a VM. I've attempted to clean up the instructions (i.e. these instructions have been updated and are not exactly what was was executed on the command line) for folks who would like to attempt the same (Note: There may be more elegant ways of setting this up :) )

# Create BMV2 package for a VM Running Debian 10

Note: "host>" is the host system which will be used to generate the Stratum
BMv2 Debian package. "stratum_dev>" is the Stratum development docker container.
"vm>" is a Debian 10 VM which will be converted into the Stratum BMv2 switch.

1. Clone the stratum repo on your host system:
 host> mkdir repos && cd repos
 host> git clone g...@github.com:stratum/stratum.git
 
2. Change dependency version for boost from libboost1.55-all-dev to
libboost1.67-all-dev (Since Debian 10 does not ship with libboost1.55-all-dev):
 host> cd stratum
 host> vim stratum/hal/bin/bmv2/BUILD

...
"libboost1.67-all-dev",
...

REF:
https://github.com/stratum/stratum/blob/main/stratum/hal/bin/bmv2/BUILD#L81-L98

3. Start the stratum docker development container and build the BMv2 package:

   host> ./setup_dev_env.sh # You should now be in the stratum container.
   stratum_dev> bazel build --stamp //stratum/hal/bin/bmv2:stratum_bmv2_debian
   stratum_dev> mkdir stratum_bmv2_configs/libs && cd stratum_bmv2_configs/libs
   stratum_dev> cp /usr/local/lib/libbmpi.so.0.0.0 ./stratum_bmv2_configs/libs
   stratum_dev> cp /usr/local/lib/libsimpleswitch_runner.so.0.0.0 ./stratum_bmv2_configs/libs

Note: The package should exist at
~/repos/stratum/bazel-bin/stratum/hal/bin/bmv2/stratum_bmv2_debian.deb on the
host. Let's move the package to a common directory:

stratum_dev> mkdir stratum_bmv2_configs/bmv2_package && cd stratum_bmv2_configs/bmv2_package
stratum_dev> cp ~/repos/stratum/bazel-bin/stratum/hal/bin/bmv2/stratum_bmv2_debian.deb ./

4. SCP the package and associated libraries to the VM.

host> ssh <user>@<VM IP>
vm> mkdir stratum && mkdir stratum/bmv2_package
host> scp stratum_bmv2_configs/bmv2_package/stratum_bmv2_debian.deb <user>@<VM IP>:/home/<user>/stratum/bmv2_package
host> scp -r stratum_bmv2_configs/libs <user>@<VM IP>:/home/<user>/stratum
host> scp -r stratum_bmv2_configs/bmv2_config <user>@<VM IP>:/home/<user>/stratum
host> scp ~/repos/stratum/stratum/hal/bin/bmv2/dummy.json <user>@<VM IP>:/home/<user>/stratum

5. Install bazel 3.7.2 on the VM
vm> sudo apt install apt-transport-https curl gnupg
vm> curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor > bazel.gpg
vm> sudo mv bazel.gpg /etc/apt/trusted.gpg.d/
vm> echo "deb [arch=amd64] https://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
vm> sudo apt update && sudo apt install bazel-3.7.2

REF:
https://docs.bazel.build/versions/master/install-ubuntu.html

3. Install dependencies and build stratum:
vm> sudo apt-get update
vm> sudo apt-get install apt-utils libjudy-dev libgmp-dev libpcap-dev mpi-default-dev libnl-3-200=3.4.0-1 libnl-route-3-200=3.4.0-1 libnl-route-3-dev libnl-3-dev libibverbs-dev libopenmpi-dev libboost-mpi-dev libboost-mpi-python-dev libboost1.67-all-dev -y

REF:
https://github.com/stratum/stratum/blob/main/stratum/hal/bin/bmv2/BUILD#L81-L98

4. Configure bmv2 associated libraries:
vm> cd stratum/libs
vm> sudo cp * /usr/local/lib/ && cd /usr/local/lib/
vm> sudo ln -s libsimpleswitch_runner.so.0.0.0 libsimpleswitch_runner.so.0 && sudo ln -s libbmpi.so.0.0.0 libbmpi.so.0
vm> sudo vim /etc/ld.so.conf
...
/usr/local/lib

vm> sudo ldconfig

5. Create links to older version of boost libraries:
vm> sudo ln -s /usr/lib/x86_64-linux-gnu/libboost_system.so.1.67.0 /usr/lib/x86_64-linux-gnu/libboost_system.so.1.62.0
vm> sudo ln -s /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.67.0 /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.62.0
vm> sudo ln -s /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.67.0 /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.62.0
vm> sudo ln -s /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.67.0 /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0

6. Install the bmv2 package:
vm> cd ~/stratum/bmv2_package
vm> sudo apt-get install --reinstall ./stratum_bmv2_debian.deb

7. Add interfaces to config file and create the empty p4_pipeline.pb.txt which
is where your pipeconfig will be stored (i.e. after being pushed by the switch):
vm> cd ~/stratum/bmv2_config
vm> cp config.txt switch_ports-spine1.txt
vm> vim switch_ports-spine1.txt

description: "Spine1"
chassis {
  platform: PLT_P4_SOFT_SWITCH
  name: "bmv2-simple_switch"
}
nodes {
  id: 1
  slot: 1
  index: 1
}
singleton_ports {
  id: 0
  name: "Eth0"
  slot: 1
  port: 0
  channel: 1
  speed_bps: 100000000000
  config_params {
    admin_state: ADMIN_STATE_ENABLED
  }
  node: 1
}
...

9. Create an empty "p4_pipeline.pb.txt" file (The P4 pipeline, pushed by the
controller, will be stored in there).
vm> vim p4_pipeline.pb.txt

10. Start the switch on port 50010:

vm> sudo stratum_bmv2 \
    --persistent_config_dir=/home/<user>/stratum/bmv2_config \
    --forwarding_pipeline_configs_file=/home/<user>/stratum/bmv2_config/p4_pipeline.pb.txt \
    --chassis_config_file=/home/<user>/stratum/bmv2_config/switch_ports-spine1.txt \
    --external_stratum_urls="192.168.122.10:50010" \
    --initial_pipeline="/home/<user>/stratum/bmv2_config/dummy.json" \
    --bmv2_log_level=info

11. Attempt to connect to the switch from the P4R shell:
host> cd ~/repos
host> git clone g...@github.com:opennetworkinglab/ngsdn-tutorial.git
host> cd ~/repos/ngsdn-tutorial
host> util/p4rt-sh  --grpc-addr 192.168.122.10:50010 --config ../stratum/stratum_bmv2_configs/bmv2_config/p4info.txt,../stratum/stratum_bmv2_configs/bmv2_config/bmv2.json --election-id 0,1

12. Use the gnmi utility to ensure that all ports are visible:
host> util/gnmi-cli --grpc-addr 192.168.122.10:50010 get / | util/oc-pb-decoder | less

Cheers!

-Syd