I cannot guess it from far away, however here are some hints:
- Try to run the simulation first with smaller amount of nodes and observe whether the memory requirement of the process is stable during the whole simulation. (i.e. try to find a small enough network that is still running on your machine). Check how much memory is required by the process. If the memory ins constantly growing no matter how small the network is then you have a memory leak bug somewhere in the code.
If you can find a working network size, experiment with the netwokr size up and down and see the changes in memory consuption. At least you would have a general idea about the actual limits...
I'm not sure about the limits, but theoretically ADHOC routing protocols are o(2) i.e. the memory consumption is quadratic as size of the routing table is proportional to the number of nodes in each node...
It could be caused also by the heavy traffic. Of you have a component somewhere which does not limit the queue length and you try to put more and more data into it, it will finally consume all memory...
So if you see ever increasing memory consuption (i.e. it never stabilizes), then I would suspect that too heavy traffic is causing the memory loss. (in this case you should configure the offending queues to start dropping packets after a while)
Rudolf