Reproducing Basic SCOPE Use Case

[Novak Boskov]

I've seen this repo (https://github.com/wineslab/colosseum-scope#scope-cellular-scenarios-for-colosseum-network-emulator), however:

- am I supposed to run `scope_start.py` on all the BSs and UEs, or only on one node?
- am I supposed to run `colosseumcli rf start ...` on my own, or can I just pass the scenario somewhere in Colosseum config?

For now, I using your `/share/nas/common/scope.tar.gz` container,  start the scenarios using `colosseumcli rt start ...`, and run `scope_start.py` on all my 4 nodes (first on the BS). But that doesn't work.

The problem is that my UE (the `srsue` program running there) never gets connected to the radios. Here is `/logs/colosseum_ue.log` on one of my UEs where I run `scope_start.py`:

```
linux; GNU C++ version 5.4.0 20160609; Boost_105800; UHD_003.009.005-0-g32951af2

Reading configuration file /root/radio_code/srslte_config/ue.conf...

Built in Release mode using commit 2a8d02c on branch master.

Opening 1 channels in RF device=default with args=default
Opening USRP channels=1, args: type=x300,master_clock_rate=184.32e6
-- X300 initialization sequence...
-- Determining maximum frame size... 8000 bytes.
-- Setup basic communication...
-- Loading values from EEPROM...
-- Setup RF frontend clocking...
-- Radio 1x clock:184.32
-- Initialize Radio0 control...
-- Performing register loopback test... pass
-- Initialize Radio1 control...
-- Performing register loopback test... pass
Waiting PHY to initialize ... done!
Attaching UE...
Setting sample rate to: 1.920000 MHz, nsamples: 98, tx_adv_usec: 51.041667

....................................
```

In what order am I supposed to run all the pieces (`scope_run.py` on all nodes and `colosseumcli rf start ...`)? Can SCOPE automate that process?

[Leonardo Bonati]

Dear Novak,

The colosseumcli command to start the scenario should be run on only a single node (any node) of the reservation (also remember to add the "-c" flag to make the scenario restart after it ends). You can then verify that the scenario started correctly with the command "colosseumcli rf info".

You can use the "scope_start.py" script on both base stations and UEs passing a configuration file to it with "python3 scope_start.py --config-file radio_interactive.conf", where the latter is the name of my configuration file. You can also pass each parameter without the need for a configuration file, the helper utility ("python3 scope_start.py --help") describes the various parameters.

As for how base stations and users are selected, you can set the number of UEs for each base station through the "users-bs" parameter. As an example, let's say users-bs = 3. In this case the first SRN will be the BS#1 and the 3 following SRNs will be three users. Then, the fourth SRN will be BS#2 and the 3 following SRNs of your reservation will be UEs, and so on.

My guess from the UE log is that the scenario is not running, or you are using a scenario that works at different frequencies than those you are transmitting at (each scenario has a center frequency, e.g., 1GHz for most Colosseum scenarios, and the channel emulator will listen at up to 80 MHz of bandwidth around the center frequency, 40 MHz per side). Another possibility is that the specific SRN you picked is not working properly (e.g., the USRP has some issues, unfortunately this happens sometimes).

Regards,

Leonardo

[Novak Boskov]

Hi Leonardo, thanks for answering!

Could you please share the containers (with the configuration files) that you used to obtain the results from your SCOPE paper (for example, Figure 6)? 
For now, I would be interested in reproducing your Rome scenario (for example, the scenario number 1018).

I still have some issues with getting it all wired up. This is what I've tried:
- My reservation has 4 nodes and each container is spawned from the same image (`/share/nas/common/scope.tar.gz`).
- SRN_1 runs `colosseumcli rf start 20039 -c` (that's the "LTE-WiFi coexistence stationary demo (4 taps)" scenario with center frequency of 1GHz).
- `colosseum rf info` suggests that the scenario is active.
- SRN_1 runs `python3 scope_start.py --config-file radio_interactive.conf`
- SRN_2 runs the same.
- SRN_3 runs the same.
- SRN_4 runs the same.

Where `radio_interactive.conf` is always the same:
{
    "capture-pkts": "False",
    "colosseumcli": "False",
    "iperf": "True",
    "users-bs": "3",
    "total-nodes" : "4",
    "write-config-parameters": "True",
    "network-slicing": "True",
    "global-scheduling-policy": "0",
    "slice-scheduling-policy": "[1]",
    "tenant-number": "1",
    "slice-allocation": "{0: [0, 5]}",
    "slice-users": "{0: [1, 2, 3]}",
    "custom-ue-slice": "True",
    "force-dl-modulation": "False",
    "heuristic-params": "{'buffer_thresh_bytes': [1000, 2000], 'thr_thresh_mbps': [0.25, 0.75]}",
    "bs-config": "{'dl_freq': 980000000, 'ul_freq': 1020000000, 'n_prb': 50}",
    "ue-config": "{'dl_freq': 980000000, 'ul_freq': 1020000000, 'force_ul_amplitude': 0.9}"
}

The TAP interfaces never get created and the UE logs are as in the previous email.

Best regards,
Novak Boskov

[Leonardo Bonati]

Dear Novak,

Please find the replies below.

Could you please share the containers (with the configuration files) that you used to obtain the results from your SCOPE paper (for example, Figure 6)? 
For now, I would be interested in reproducing your Rome scenario (for example, the scenario number 1018).

The "scope" container that is publicly available on Colosseum can be used to run that specific experiment. I do not have the exact configuration files I used at the time, but you can start from the radio_interactive.conf file and tune the parameters accordingly. For instance, if you have 3 users, set "users-bs": "3", if you are not interested in network slicing functionalities, set "network-slicing": "False". At that point, you can safely ignore the remaining parameters related to network slicing.

 

I still have some issues with getting it all wired up. This is what I've tried:
- My reservation has 4 nodes and each container is spawned from the same image (`/share/nas/common/scope.tar.gz`).
- SRN_1 runs `colosseumcli rf start 20039 -c` (that's the "LTE-WiFi coexistence stationary demo (4 taps)" scenario with center frequency of 1GHz).
- `colosseum rf info` suggests that the scenario is active.
- SRN_1 runs `python3 scope_start.py --config-file radio_interactive.conf`
- SRN_2 runs the same.
- SRN_3 runs the same.
- SRN_4 runs the same.

 It looks like you are using an undocumented scenario, which is currently being tested by people at Northeastern. I would suggest using one of the scenarios that is documented on freshdesk (the Rome static scenario is 1017 for instance):

- https://colosseumneu.freshdesk.com/support/solutions/articles/61000276224-scenarios-summary-list

- https://colosseumneu.freshdesk.com/a/solutions/articles/61000295793

 

Also, make sure to add the "-c" (cyclic) flag to keep the scenario running (e.g., "colosseumcli rf start 1017 -c"), otherwise it may happen that the scenario ends before you actually start transmitting. Because of how Colosseum works, transmissions are only possible when a RF scenario is active, and the testing scenario you were using only lasts a single second. My guess is that by the time your nodes started transmitting, the scenario had already ended.

Where `radio_interactive.conf` is always the same:
{
    "capture-pkts": "False",
    "colosseumcli": "False",
    "iperf": "True",
    "users-bs": "3",
    "total-nodes" : "4",
    "write-config-parameters": "True",
    "network-slicing": "True",
    "global-scheduling-policy": "0",
    "slice-scheduling-policy": "[1]",
    "tenant-number": "1",
    "slice-allocation": "{0: [0, 5]}",
    "slice-users": "{0: [1, 2, 3]}",
    "custom-ue-slice": "True",
    "force-dl-modulation": "False",
    "heuristic-params": "{'buffer_thresh_bytes': [1000, 2000], 'thr_thresh_mbps': [0.25, 0.75]}",
    "bs-config": "{'dl_freq': 980000000, 'ul_freq': 1020000000, 'n_prb': 50}",
    "ue-config": "{'dl_freq': 980000000, 'ul_freq': 1020000000, 'force_ul_amplitude': 0.9}"
}

Please note that node 1 is the base station, so the slice-users should be set to  "{0: [2, 3, 4]}". Also, I believe the "total-nodes" parameter is not used anymore.

 

The TAP interfaces never get created and the UE logs are as in the previous email.

 My mistake, there is no tap interface involved, only the tun interfaces are created (e.g., srs_spgw_sgi and tun_srsue). If the UEs successfully connect, they should use these interfaces to exchange traffic with iPerf3 if you enable the related parameter in the configuration file ("iperf": "True").

Best,

Leonardo

[Novak Boskov]

Thanks, Leonardo!

If "total-nodes" in the SCOPE config is now ignored, how do I set the number of nodes?
What is the "Number of Nodes" parameter in each scenario? If the Rome scenario has "Number of Nodes = 50", does that mean that my reservation needs to have exactly 50 nodes?

Best regards,
Novak Boskov

[Leonardo Bonati]

Hi Novak,

The total number of nodes will be derived from your reservation details. The nodes ping themselves on the col0 interface to find out the number of active nodes. Alternatively, you can enable the colosseumcli flag in the config file and they will use this toolkit instead to find out the total number of nodes (this is faster for interactive reservations).

You can check the maximum number of nodes each scenario supports (you can also use the scenario with fewer nodes) in one of the following ways:

- checking the freshdesk links in my previous message

- running the "colosseumcli rf scenario list" (or similar, I'm going from memory here, "colosseumcli --help" should return the exact commands)

- once the scenario is active, with the "colosseumcli rf scenario nodelist" / "colosseumcli rf radiomap) (again, from memory, "colosseumcli --help" should return the exact commands. The following page also describes colosseumcli https://colosseumneu.freshdesk.com/support/solutions/articles/61000253397-colosseum-cli)

Regards,

Leonardo

--
You received this message because you are subscribed to the Google Groups "Colosseum Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to colosseum-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/colosseum-users/a7ed159e-edad-4407-a784-26e52864547bn%40googlegroups.com.

[Novak Boskov]

Hi Leonardo,

I've tried a very basic setup closely following your instructions and it doesn't seem to work.

I have a reservation with 4 nodes only. The modification of `radio_interactive.conf` is as follows:
```
{
  "capture-pkts": "False",
  "colosseumcli": "True",

  "iperf": "True",
  "users-bs": "3",
  "total-nodes" : "4",
  "write-config-parameters": "True",

  "network-slicing": "False", 
  "global-scheduling-policy": "0",
  "slice-scheduling-policy": "[1, 1, 1]",
  "tenant-number": "3",
  "slice-allocation": "{0: [0, 5], 1: [6, 11], 2: [12, 16]}",
  "slice-users": "{0: [3, 6, 9, 14, 17, 20, 25, 28, 31, 36, 39, 42], 1: [4, 7, 10, 15, 18, 21, 26, 29, 32, 37, 40, 43], 2: [2, 5, 8, 11, 13, 16, 19, 22, 24, 27, 30, 33, 35, 38, 41, 44]}",

  "custom-ue-slice": "True",
  "force-dl-modulation": "False",
  "heuristic-params": "{'buffer_thresh_bytes': [1000, 2000], 'thr_thresh_mbps': [0.25, 0.75]}",
  "bs-config": "{'dl_freq': 980000000, 'ul_freq': 1020000000, 'n_prb': 50}",
  "ue-config": "{'dl_freq': 980000000, 'ul_freq': 1020000000, 'force_ul_amplitude': 0.9}"
}

```
As you said, "network-slicing": "False" should make the rest of the configuration ignored.

I do the following steps:
1. On node 1: `colosseumcli rf scenario nodelist 1017 -c` (that's the scenario you suggested, with the `-c` option)
2. On nodes 1..4: `cd /root/radio_api/ && python3 scope_start.py --config-file radio_interactive.conf`
3. On nodes 1..4: `tmux a -t scope`

The results are:
- Node 1 (the only base station) connects to the radio software successfully and starts the `iperf` server.
- Node 2 (one of the UEs) also connects to the radio software and successfully instantiates its `iperf` clients. Now one node talks to the base station.
- Nodes 3 and Node 4 never connect to the radio software and their `iperf` clients return "error - unable to connect to server: Network is unreachable".
- `colosseumcli rf info` suggests that the radio scenario is active and "scenario_cycle" is "True".

What I'd expect is that nodes 2 to 4 connect to the radio software and obtain their IPs. Then `iperf` would work.

Without the artifacts you used in your SCOPE paper, it's hard to get anywhere close to the results you reported in the paper.

[Leonardo Bonati]

Dear Novak,

I would say that this is good progress. You managed to establish connectivity between a base station and one UE, and also to exchange iPerf traffic between the two. This also means that the Colosseum scenarios started properly. Now the thing left to do is to check what is going on with the other 2 UEs. Checking the logs of the script and the configurations of base station and UEs might help in this. These configurations are in the srslte_config and in the scope_config directories.

If nodes 3 and 4 are not connecting to the USRPs, I would try making a new reservation, and using other SRNs as sometimes the USRPs get stuck. Since iPerf is running in loop at the UEs, you will get the network error until the UEs connect to the base station (which never happens for UEs 3 and 4 in this case).

Few other things to look at:

- check that the UEs are all assigned different IMSI values and that these values are in the database used by the EPC (this should be done automatically by the startup script if all the SRNs are starting the same container)

- check that you don't have stale configuration files in the scope_config directory. There is a script to remove old files (if any): remove_experiment_data.sh

- check that all the nodes are using the same configuration file, and that the nodes discover the correct number of nodes (this can be read in the logs of the python script to start base station and UEs)

As a minor note, the command "colosseumcli rf scenario nodelist 1017 -c" should be split in two:

- "colosseumcli rf start 1017 -c" to start the scenario, and

- "colosseumcli rf scenario nodelist" to check the nodes in the scenario

Best,

Leonardo

To view this discussion on the web visit https://groups.google.com/d/msgid/colosseum-users/f70cb400-6f1c-485b-9f4a-a94a04a26d15n%40googlegroups.com.

[Novak Boskov]

Hi Leonardo, 

Hm... That's pretty much me setting up srsLTE myself. Wasn't SCOPE meant to automate that process by handling the configuration?

It seems like it requires a fair amount of manual interventions for even the simplest cases.

Could you clarify what you mean by "using other SRNs"? Do you mean using different containers? There is only one `scope` container available to Colosseum users, as far as I can tell. It is the one in `/share/nas/common/scope.tar.gz`.

> "colosseumcli rf scenario nodelist 1017 -c" should be split in two

Yep, that was my typo in the email. :)
As I said, `colosseumcli rf info` suggests that the radio scenario is active and "scenario_cycle" is "True".

I've submitted an office hours request via freshdesk, our team would appreciate if you could demonstrate the configuration process for one of the experiments in your SCOPE paper. It can be any of Rome, Boston, or Utah.

Best regards,
Novak Boškov

[Novak Boskov]

Hi Leonardo,

Are there any updates?

My basic questions are still the same:
- Which container image (SRN) are you most confident in?
- Which configuration files are you most confident in?

I see you updated your `/share/nas/common/scope.tar.gz` yesterday. The default SCOPE config file have been slightly changed (for example, `colosseumcli: "True"` now). Are these changes significant?

Unfortunately, it seems like the reservation plays an important role in all this. In some reservations, I manage to get two UEs talk, in others they just never successfully connect to USRPs. With exactly the same containers and exactly the same config files. This puzzles me. :)

I've checked all that you've suggested. Concrete actions are below.

- "check that the UEs are all assigned different IMSI values and that these values are in the database used by the EPC"

Yes, UEs do get different IMSIs (as per `/root/radio_code/srslte_config/ue.conf`) and these in the database (`/root/radio_code/srslte_config/user_db.csv`). Although, I don't see any IMSI-related parts in `/root/radio_code/srslte_config/epc.conf`.

-  check that you don't have stale configuration files in the scope_config directory

No, there are no stale files. I clear them before each run.

- "check that all the nodes are using the same configuration file, and that the nodes discover the correct number of nodes"

They do discover the right number of nodes. Yes, I use the same container for all 4 nodes. But, per your `scope_run.py` script, you don't use radio interfaces to discover them unless `colosseumcli: "True"` in the SCOPE config file.

Best,
Novak Boškov

[Leonardo Bonati]

Novak,

I have thoroughly tested the containers and scripts and they work as expected. Just to give you a quick recap, these are the steps you need to execute.

1. Make a reservation with 4 SRNs with the "scope" image in the "common" section of Colosseum. (Make sure you don't have an image with the same name on your team's directory, otherwise this image will be picked up instead. The slight updates you noticed were made in an effort to provide users with a more straightforward image to use. They are not significant but I would suggest starting from this image.) You don't need to modify any parameter of the basic image, and you don't need OctoClocks in your reservation.
2. once the reservation starts, SSH on the 4 SRNs (let's call them SRN1-4)
3. on SRN1 execute: colosseumcli rf start 1017 -c
4. on SRN1 execute: cd /root/radio_api && python3 scope_start.py --config-file radio_interactive.conf
5. on SRN2 execute: cd /root/radio_api && python3 scope_start.py --config-file radio_interactive.conf
6. on SRN3 execute: cd /root/radio_api && python3 scope_start.py --config-file radio_interactive.conf
7. on SRN4 execute: cd /root/radio_api && python3 scope_start.py --config-file radio_interactive.conf
8. once the above script ends, execute the following on all the SRN to see the programs running sessions: tmux a

After doing this, users connect to the base station and start exchanging traffic generated through iPerf.

The above commands are all you need to use the SCOPE image on Colosseum. Of course you can use this as a starting point and add novel capabilities according to your research needs on your own time. You can also contribute back if interested in doing so.

Unfortunately, being hardware devices, it may happen that USRPs do not work as expected because of normal wear and tear. Please note that these issues do not depend on the specific protocol stack in use (e.g., SCOPE in this case). If this happens, please try starting a new reservation, which should allocate you a new set of SRNs.

Regards,
Leonardo

To view this discussion on the web visit https://groups.google.com/d/msgid/colosseum-users/44c0836b-fa2d-405d-ac65-bd30780f806an%40googlegroups.com.

[Novak Boskov]

Thanks Leonardo,

This seems to work, except when an USRP is not functional. For example, whenever my reservation gets `sync-edge-036`, it never manages to attach to its USRP (due to a radio link failure). And that happens very often.

Found Cell:  Mode=FDD, PCI=1, PRB=50, Ports=1, CFO=0.3 KHz
Setting sample rate to: 11.520000 MHz, nsamples: 350, tx_adv_usec: 30.381944
Found PLMN:  Id=00101, TAC=7
Random Access Transmission: seq=14, ra-rnti=0x2
Random Access Transmission: seq=25, ra-rnti=0x2
Random Access Transmission: seq=0, ra-rnti=0x2
Random Access Transmission: seq=7, ra-rnti=0x2
Random Access Transmission: seq=4, ra-rnti=0x2
Random Access Transmission: seq=28, ra-rnti=0x2
Random Access Transmission: seq=40, ra-rnti=0x2
Random Access Transmission: seq=35, ra-rnti=0x2
Random Access Transmission: seq=40, ra-rnti=0x2
Random Access Transmission: seq=43, ra-rnti=0x2
Warning: Detected Radio-Link Failure

Just a wild guess, is USRP functionality related to the color of rows on the reservation interface (screenshot attached)? 

[Leonardo Bonati]

Hi Novak,

I'm glad to hear things are working now.

The color of the rows is not an indication of the USRP functionality but rather of which SRNs (i.e., a pair of server + radio) are currently available. In general, the meaning is as follows:

- white row: the SRN is in the production environment and can be booked by users. Among these:

    + grey: the SRN is currently in a reservation made by another team

    + blue: the SRN is currently in a reservation made by your team

- red: the SRN is out of the production environment. This may happen when the SRN is in a sandbox environment to test new functionalities, or the SRN is in maintenance mode (e.g., the USRP or server are being repaired, issues with networking, etc.).

Regards,

Leonardo

To view this discussion on the web visit https://groups.google.com/d/msgid/colosseum-users/fff62469-8d59-4242-88f7-3a725d39d9cdn%40googlegroups.com.

[MUHAMMAD ENAYETUR RAHMAN]

Hello Novak,

Did you try with multiple BS and multiple UEs?

I observed some of the UEs are not transferring data saying: unable to reach the server.