NS3 Specify Threads

[Michael Byers]

Hello all,

We're doing some research on the scalability of NS3 regarding some different factors in comparison to another network simulator. 

Is there a way to specify the number of worker threads that NS3 creates for a specific test?

[Tommaso Pecorella]

Hi,

as far as I know, there's only one thread, unless you use MPI. MPI itself is supported by ns-3, but it's a bit more complex than just a thread. I'd suggest to check the manual page:

https://www.nsnam.org/docs/models/html/distributed.html

Have fun,

T.

[Michael Byers]

Thank you, I was unaware of the MPI implementation. However, running htop I can see there are multiple threads running at once for this one process of my test. Is it known if there's way to set a limit / lock on the number of threads that get created?

[Tommaso Pecorella]

If you're referring to the "test.py" script, it's like a compilation script: it will launch a number of simulations in parallel (depending on how many processors are detected).

This, of course, doesn't really have anything to do with a real simulation. You can launch 20 simulations in parallel and your O.S. will manage them, but you're not multi-threading.

Have fun,

T.

[Michael Byers]

By test, I really just mean my own NS3-DCE script that I've written to simulate a large (100-1000+) network of iperf clients/servers. But thank you for your help, so you're saying there really isn't a way to do what I'm asking outside of using MPI?

[Tommaso Pecorella]

I'm not really sure of what you're trying to do tbh.

Still, the answer is probably that you'll need MPI. The reason is that normal threads can't really help in a discrete event simulator, unless you have a look-ahead scheduler able to forecast if one event will have a side-efect on th status of the simulator that, in turn, will affect the existing event in execution (and, in case things go wrong, will trash "dirty" events).

This is somewhat mitigated by the MPI module restrictions. if you study the documentation you'll understand why threads aren't used (and they shouldn't be used either at event-level).

You can, of course, have a multithread part in an implementation - e.g., a Dijiskra algorithm could be implemented in multi-thread,but that's an implementation details and it doesn't really affect or involve the general ns-3 structure.

Summarizing...  you could have multi-threaded parts, but not in the general scheduler (but there you can use MPI).

Hope this helps,

T.

[Rob Jansen]

I completely disagree with this answer.

The same way that work can be split upon multiple processes with MPI, they could be instead be split among multiple threads. As long as the work (events) are properly separated among work units (e.g., virtual hosts), and dependencies are properly respected, this work can be split and executed in any processing unit, be that a separate process or a separate thread.

This is in fact how the Shadow network simulator works, and I can attest to the fact that multiple threads do indeed dramatically improve runtime performance.

https://shadow.github.io/

Anyway, I believe the question was what NS3 is actually doing, not what it could do. Still, the user noticed multiple threads running in top, so it seems like NS3 is utilizing threads in some way. I have no experience, so I cannot answer that.

[Tommaso Pecorella]

Hi Rob,

this discussion is or sure fascinating, but it could go far beyond the original thread question.

Just a personal (and small) note to clarify to the future readers.

Yes, multi-threading can improve dramatically any program performance IF you can execute parallel threads. This is true for any program, not only network simulators.

Most ns-3 code should be thread-safe, but I haven't seen multi-threading being actively used in the code.

The main problem is about the object status and how locks are used to... lock the objects status if more than one thread is changing them.

Now, in a DE simulation, you can execute threads:

1) in the same event

2) between different events

In the first case there's no problem. it's up to the developer to do it. It is neither suggested or forbidden by ns-3, and most classes are thread-safe.

In the second case there IS a problem, because different events are (usually) belonging to different times. If you have event A and event B scheduled for time Ta and Tb, and you execute them in two different threads, then you must be sure that the global system status observed by event B is not changed by the execution of event A. This is, basically, why MPI in ns-3 is subject to some restrictions (see the MPI manual section).

Again, it's not a problem of "it's useless" or "it's impossible", it's a problem of "it's damn complex".

If someone would be so nice to work on it, it would be a welcome addition for sure. However, the profiling data suggests that the real bottlenecks in ns-3 are elsewhere, and that threads aren't our first priority (note: other people in the ns-3 community can have a different opinion). This (again, it's a personal point of view) is even more important because one simulation doesn't mean anything for most researchers. I need to run Monte-Carlo simulations, so multi-threading is [kinda] useless. My simulations are CPU-bound, and if I have 16 cores, I launch 20 simulations at once. Will multi-threading give some more speed ? Not really.

Anyway, the discussion is interesting but is going off-topic. My suggestion is that if there's enough interest in going multi-thread we should:

1) identify the use-cases

2) move the discussion to ns-dev mailing list (or also there)

3) write down a work plan

Cheers,

T.

[Madan Pande]

Hi,
          Your arguments would not cut much ice in discrete event simulation. I have some experience running  LTEAdv Carrier Aggregation with and without threads, the results while very interesting are at best comparable only. Perhaps, let me support what Tom is Trying to convey...albeit within my limited experience compared to the Gurus of this Simulator...
2. The Simulator Entity, only processes what is in the Event-Queue (s), it is a resource a thread will have to lock and unlock, however you may split the code, before you join the threads, say in a boss-worker model.

I hope the msg is clear...

With Regards,
madan

---

Date: Tue, 11 Aug 2015 07:21:46 -0700
From: rob.g....@gmail.com
To: ns-3-...@googlegroups.com
Subject: Re: NS3 Specify Threads

--
You received this message because you are subscribed to the Google Groups "ns-3-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ns-3-users+...@googlegroups.com.
To post to this group, send email to ns-3-...@googlegroups.com.
Visit this group at http://groups.google.com/group/ns-3-users.
For more options, visit https://groups.google.com/d/optout.

[Thomas Burger]

Hi,

just a short comment because I read that you are using DCE. DCE actually uses threads, but only for virtualization reasons. ns-3 DCE allows to integrate existing "real-world" applications and protocols, which are usually running as one (or more) POSIX process(es) on your computer.

In the default configuration, ns-3 DCE handles each POSIX process of each node as one pthread. However, these threads are executed sequentially. At first sight, this maybe somehow strange because you have multiple threads but they are not executed parallel. You can easily see this with a modified version of the dce-sleep example (see also attached files):
- ns-3-dce/myscripts/sleep/tenseconds.c: Make application run forever, put a while(1) around the sleep(10)
- ns-3-dce/myscripts/sleep/dce-sleep.cc: Increase number of nodes and simulation runtime in simulation setup
Now, when running the simulation, you can see in htop that 500 threads (= number of started applications = number of nodes) have been created, but the simulation only utilizes 100%, i.e. one core.

Further notes:
- Instead of pthreads, ns-3 DCE can also be run with ucontext fiber manager. For both variants, please have a look at the DCE manual (www.nsnam.org/overview/projects/direct-code-execution), the paper from Hajime Tazaki et al ("Direct Code Execution: Revisiting Library OS Architecture for Reproducible Network Experiments", https://hal.inria.fr/hal-00880870/), or Mathieu Lacage's PhD thesis ("Experimentation Tools for Networking Research" http://www.cutebugs.net/files/thesis.pdf).
- I can't tell whether the existence of multiple threads/fibers in DCE would simplify parallel execution or not. For me, it's more important that ns-3 DCE combines the benefits of emulation (execution of existing applications and protocols) and simulation (reproducibility, debuggability, time dilation, ...). See again mentioned literature for more information. As Tommaso pointed out, you can (and should) still run multiple simulations in parallel.
- Finally, please note that there is also an example with MPI and DCE (ns-3-dce/myscripts/dce-mpi-udp), but I haven't looked at it yet.

Best regards,
Thomas

PS: It would be nice if you inform this newsgroup about the results of your research :-)

[Luong Doanh]

Hi all,

I have problems of using threads in NS3 DCE.

Currently, I am trying to implement the Kodo-RLNC library in NS3-DCE. Our Ns-3 simulation setup consists of two nodes: one UDP sender and one UDP receiver communicating via a GEO satellite path with propagation delay of 340 ms. The sender sends the packets, the receiver receives the packets and sends back the SNACK.

The situation is that the sender sends the packets, the receiver receives the packets and sends back the snack, but the sender can not receive the snack. As in my understanding, the problem is that NS3 – DCE use multiple threads for the applications. In our case, we have two threads, one for the sender and one for the receiver. However, NS 3 – DCE manages threads in sequential manner. Then, in our case, the thread for the sender starts, then the thread for the receiver starts. 

The solution could be two alternatives:

- Using MPI to support parallel running of the applications with asynchronous operations. 

- Using multiple threads at the sender with synchronous operations: one thread for sending, and one thread for receiving.

However, I tried both ways but the problem is still the same. 

Note that, the performance of sender and receiver are very good outside of NS3-DCE. I compiled two binary: sender, receiver, and running them in 2 different terminals. The communication between them was as expected.

Could you give me some suggestions on how to using MPI, or some other alternatives?

Best regards.

Doanh

Vào 21:46:04 UTC+2 Thứ Bảy, ngày 15 tháng 8 năm 2015, Thomas Burger đã viết: