Memory Leak Running NS-3 Scenarios in Loop (wh Pyton Bindings)

[Shyam Parekh]

I'm using NS-3 with Python Bindings. I've a function, say network_step(), that runs an NS-3 scenario, and it ends with ns.Simulator.Destroy(). network_step() is called from main in a loop. I find that memory consumption keeps on increasing while progressing through the loop. I don't understand why this happens. I thought that ending each NS-3 run with ns.Simulator.Destroy() would release all the memory associated with that run. Please let me know if you've any insight on why this happens, and how to remedy it. Thanks!

Just to be clear, my set up is as follows:

N = 100000

for i in range(N):

      network_step()

def network_step():

        ....

        Python script for an NS-3 scenario

        ....

        ns.Simulator.Stop(ns.core.Seconds(simTime))

        ns.Simulator.Run()
        ns.Simulator.Destroy()

[Gabriel Ferreira]

Either a memory leak inside the C++ code or on Python. Do you define any global scope variable inside network_step?

[Shyam Parekh]

Yes, I do. I've some extern variables defined in the Python script (using cppyy.cppdef) and also in the C++ codes (added for custom modeling).

Thanks Gabriel!

[Gabriel Ferreira]

If you have a global variable defined via cppdef (e.g. to capture some statistic), than you need to clean it up between runs.

Same for the python global variables.

If you don't see anything that should be leaking, I'd say to give scalene a try. https://github.com/plasma-umass/scalene

[Shyam Parekh]

Thanks Gabriel! I'm not sure how exactly to perform clean up.

For the following in network_step()

 ns.cppyy.cppdef(
 """
 namespace ns3 {
         extern double meas[5];
 } // namespace ns3
 """
 )

I tried using "del ns.cppyy.gbl.meas" at the end of network_step(). But, it didn't work.

Can you please point me in the right direction?

Thanks,

Shyam

[Gabriel Ferreira]

extern double => values are allocated elsewhere. They technically can't leak from that.

Use scalene to profile your script. Just one or two iterations should be enough.

[Shyam Parekh]

Thanks Gabriel! I've been playing around with Scalene over the weekend. Following is screenshot from a recent short run. It does show memory growth, but I have not been able to identify the cause. Any suggestions?

[Gabriel Ferreira]

Looks fine to me. This means that if there is a leak, it really is on the C++ part.
Now you need to use a different profiler, called Heaptrack. https://www.nsnam.org/docs/manual/html/profiling.html#id3
It will track if allocated memory blocks were freed by C++.

[Shyam Parekh]

Thanks Gabriel!

[Shyam Parekh]

Hi Gabriel,

I attach a simple example with significant memory leak. I'm just running lena-simple-epc with Python bindings 2000 times in a loop. I've set numNodePairs = 20.  I'm attaching the Python script file. Although I'm changing the seed for each of these 2000 runs, I think these are probably identical runs as there is no randomness.

The following screenshot from Scalene shows a steady memory build up and suggests a possible leak.

Also, after running the loop only 5 times, heaptrack shows a large number of leaked allocations. A more detailed analysis offered by heaptrack doesn't seem to offer any clear indication about where these leaked allocations are coming from. Maybe that's due to Python bindings.

Please let me know if you've any suggestion for eliminating the memory leak.

Thanks,

Shyam