NETCONF with ns-3

[Bill Tao]

Hi. Happy New Year!

So I have been working on an SDN optical module in ns-3. At first, I simulated the SDN capability with a basic implementation of a controller entirely within ns-3. Now I am looking at expanding my implementation so it can interact and accept control from real SDN software, external to the ns-3 space. In particular, I am looking at NETCONF which is a common candidate for SDN optical networks. And I would like to seek some advice on my ideas.

I imagine the following implementation structure.

                     NETCONF client / SDN controller (external)

                                               ^

                                                |          

                                ____________________

                                |         Network (ns3)  |

                                |____________________|

                                                |

                                               V

                            NETCONF server/ SDN agent (external)

                            ---------------------------------------------------------------

                                  Optical Network Device (ns3)

The NETCONF client and server are software external to ns3, they are the entity that sends and receives the SDN control. I wish to transport this control signal using simulated network in ns3. And when the NETCONF server receives the control, it configures the underlying device, which has already been simulated by ns3 objects.

Now the NETCONF protocol uses a SSH transport layer. This implies the use of Tap bridge to direct the control message into ns-3 simulated network (Approach 1). However, when it comes to interfacing NETCONF server/agent with the simulated network device, it makes more sense to use the approach taken by the OFSWITCH13 module: including the external software as a library and making direct calls to some overridden function outside ns-3 (Approach 2).

Now enters the realm which is really fuzzy to me. (I may be using lots of wrong terms here so please correct me). My question is, for approach 2 everything is still on a single thread in sequential function calls, and for approach 1 ns-3 on one thread just for packet transportation and the external software is on another listening on the incoming packets, so is it possible at all to create an implementation that is both approach 1 and approach 2? That is, is it possible to interact ns3 with the same piece of external software via both tap bridge and direct function call?

Or, how would you go about this implementation?

Thanks a lot for any input!

[Bill Tao]

Hi.

After some more investigation, it seems a possible approach for me would be to create an ns-3 callback for the external program (so the callback would be cross-thread too).

Is this possible to do? Are there any guidelines or examples I could follow?

Thanks!

[Tommaso Pecorella]

Interesting questions - but I have no clear answer. I'd check the OpenFlow 1.3 module tho.

[Tom Henderson]

Bill, inline below.

On 1/23/23 16:18, Bill Tao wrote:
> Hi.
>
> After some more investigation, it seems a possible approach for me would
> be to create an ns-3 callback for the external program (so the callback
> would be cross-thread too).
>
> Is this possible to do? Are there any guidelines or examples I could follow?

The most recent examples of interactions between external programs and
ns-3 that I'm aware of are probably the AI/ML frameworks for ns-3:

ns3-gym:
https://apps.nsnam.org/app/ns3-gym/

ns3-ai:
https://apps.nsnam.org/app/ns3-ai/

Another example in the existing codebase is the support for the PyViz
visualizer, which allows the GUI to interact and control the execution
of the ns-3 program (via a VisualSimulatorImpl specialization of the
SimulatorImpl class).

More below...

> lots of wrong terms here so please correct me). _My question is,_

> for approach 2 everything is still on a single thread in sequential
> function calls, and for approach 1 ns-3 on one thread just for
> packet transportation and the external software is on another

> listening on the incoming packets, _so is it possible at all to

> create an implementation that is both approach 1 and approach

> 2?_ That is, is it possible to interact ns3 with the same piece of

> external software via both tap bridge and direct function call?

I don't see why you couldn't use both approaches. Generally the Tap
bridge is for interacting with external processes that are running in
real time; ns-3 either has to be able to read/interpret the packets to
interact with the outside world, or it has to provide transport (sending
packets through a simulated topology without trying to read their contents).

If your application uses SSL transport, and you want to read these
packets in ns-3, you would need SSL support in ns-3 (which I'm not sure
whether anyone has tried).

>
> Or, how would you go about this implementation?
>

Another SDN approach that Jared Ivey followed a few years ago was to use
Direct Code Execution to support the controller directly in ns-3 DCE.
Most controllers are written in Java or Python, while ns-3 DCE is for
C/C++, but Jared was able to build DCE versions of Python and Java and
use those to run the controllers. Unfortunately, I don't think he left
any software artifacts about this, but you can read about this approach
here:

https://dl.acm.org/doi/10.1145/2915371.2915383

Hope this helps,
Tom

[Bill Tao]

Thanks, Tom and Tommaso,

These are very helpful information!

At the moment, I am trying to have a proof-of-concept setup with just an ns-3 simulated transport network between my server and client. I managed to bind the server-side socket to an ns-3-created tap device, by directly using the demo server implementation from the NETCONF library. However, I am not sure yet how I should do this on the client side. From my understanding, I can set up a virtual container and bridge it to a tap device into ns-3. Only this requires building the library again from within the container which I prefer not to do at the moment (because that comes with additional steps e.g. setting up internet for the container which seems like a hassle atm). 

I also read about the FdNetDevice class, which claims "can be associated to a TAP device, to a raw socket, to a user space process generating/consuming traffic, etc". But I am not able to find an example. Any chance this can help with my situation here?

The DCE also looks very promising, even without the work by Jared as the NETCONF library I use is built on C/C++. A question on this would be how compatible it is with ns-3.36 which is the version I am using?  I can't find compatibility info on its page but the intro does use ./waf instead of ./ns3.

Thanks again for all your help!

[Bill Tao]

Some updates and a follow-up question here.

I managed to get some progress with what I want to do here. I have got ns-3 to provide transport between the controller and the agent. Now I want to get the agent, which is running as the external program on a different process, to tap back into ns-3 to configure the ns-3 simulated network devices. To do this, I am currently following the implementation of tap-creator.cc and tap-bridge.cc to enable interprocess communication via unix socket, and it's looking promising.

The follow-up question: 

I am using fork() and execlp() to spawn the externa program as a child process. The spawning part seems to work well, but at the end of the simulation, when it hits Simulator::Destroy(), the child process is still running (the child process run indefinitely with a while loop). And I have to kill from the OS. My question is, is there any ns-3 way to clean up the child process when the main ns-3 thread exits?

Once again thanks for all your help, it's been really helpful.

[Tommaso Pecorella]

Hi,

I'd say to kill the forked process by using kill, see for example https://stackoverflow.com/questions/6501522/how-to-kill-a-child-process-by-the-parent-process

However, I'm not totally sure about what execlp does w.r.t. the pid of the process. Try...

[Bill Tao]

Hi, all.

It's been a while but I am coming back to working on this again. I want to augment my setup and again would love to receive your input on this.

I was able to get my original setup working thanks to the pointers from Tom and Tommaso. That is, I was able to get an SDN controller (inside a LXC) to communicate with an SDN agent (spawn in native Ubuntu user space). I spawned the SDN agent natively so it can initiate and receive function calls with an ns-3 simulated device class, as a way to impose control from the control plane to the data plane in an SDN way. This setup is illustrated in the figure below.

Now the augmentation I want to make is essentially to spawn multiple instances of the SDN network devices and their accompanying SDN agents (illustrated by the next figure). The issue I encounter is that the NETCONF agent I am using does not natively support multiple instances on one machine. So I need a way to isolate each NETCONF agent instance (and its datastore (sysrepo)). For this, I am thinking of putting each agent instance into an LXC. However, this would then stop me from being able to make direct function calls between the simulated network device class and the SDN agent. 

From my research, it seems that I would need to resort to the remote procedure calls to make this interface work again between the two entities. Currently, I am learning about grpc and assessing its suitability. But again, I am exploring at the edge of my software knowledge here so any pointers, suggestions, examples would be super helpful and highly appreciated! Perhaps I am overcomplicating? Let me know.

Thank you!

Best regards,

Bill