Multiple simulation runs Problem

[Jack Higgins]

Hi all!,


I am trying to implement an example with multiple simulation runs with different attributes (data rate, num nodes etc)
for a protocol I implemented called MASP in the same example file,  nothing out of the ordinary.

However, I noticed that whenever if  I set multiple simulation runs in the same example the Rx packets changed
compared to running each simulation individually (using the same attributes of course )




For example using  AODV protocol with 120 nodes:

Multiple Simulation
2 Runs in  with application  data rate (90000bps and 10,0000bps)
TotalTx (pkts)    TotalRx (pkts)
1088                     207
1221                     220


Single Simulation
 1 Run with 10, 0000 bps
TotalTx (pkts)    TotalRx (pkts)
1221                    163


My guess is that somehow the next simulation is starting before the current simulation ends which in the end affects the Rx packets. That or something related to the seed???, or maybe my implementation of making multiple experiments is incorrect.
Whichever is the case, can somebody point me in the right direction?

I checked  this post https://groups.google.com/forum/#!searchin/ns-3-users/multiple$20experiments/ns-3-users/08MrjCmjVxQ/fuoSUfJGAwAJ  of somebody having a similar issue. But no solution was provided.

I attached an example of my implementation. I just commented the parts related to my protocol but the example have the same issues with the AODV protocol.

[Jack Higgins]

Ok the issue is related to the "Uniform Random Variables" and their stream.

I fixed the stream in the example for the RandomRectanglePositionAllocator so the positions of the nodes wont change among simulations with the same attributes. That fix the issue of nodes positions changing.

Now is necessary to fix the random variables inside the protocol...... currently testing with AODV to see if I can obtain the same results with multiple runs.... probably  AssignStream in the AODV protocol will be necessary.

[Tommaso Pecorella]

Hi Jack,

the analysis is correct, meaning that the random stream is not reset between successive runs, unless you set manually the seed (which I wouldn't really suggest).

This is the intended behaviour. In this way, for example, you can run multiple times the same experiment and have "different" results (i.e., equivalent to the ones you'd have if you run the experiment multiple times changing the seed each time).

Cheers,

T.

[Jack Higgins]

As always Thank you for your reply Tommaso.
But After reading for the 3rd time the section in the manual regarding Random Variables and some other post I am still a little bit confused.
From what I understand in your post,  it is intended to have different results even if I set an specific seed with an specific seed run and assign an specific stream using the same attributes.  ( Shouldn't this ensure an specific result every time?)

All this because the random stream is not reset between successive runs , correct?

This behavior , in a way replicate an scenario of the real life where all the variables are never the same and results change between "each time".
But for the sake of experiment purposes shouldn't be possible to obtain the same results, regardless running with successive runs or not? (as long as I use the same attributes, seed, stream etc)



In the example I posted before with AODV  I modified the example to provide the RandomRectanglePositionAllocator with an specific stream. 
This ensure  in my example that I will have the same topology as long as use the same number of nodes.

This, however will not ensure to have the same results(number of Rx packets) if I run the same experiment multiple times in the same file.


 test.CaseRun(120,heightField,widthField,speed,1024,10000,0.4,AODV,false,false,PRINTFILE); // Rx packets  = 55
 test.CaseRun(120,heightField,widthField,speed,1024,10000,0.4,AODV,false,false,PRINTFILE); // Rx packets = 35

(both should result in Rx = 55)


Sorry if is too long  Tommaso or I am just repeating  the same thing.... Thank you for you support.

[Tommaso Pecorella]

Hi,

better a long post than a confused one :)

You're right, and the difference could be from:

1) bugs

2) Didn't you run a different simulation before the one you're considering (in the multi run I mean) ?

Cheers,

T.

[Jack Higgins]

Hi tommaso thanks again,

About bugs.... Not sure... I havent seen other experiments in AODV like this one to have a way to compare. About 2) , I only ran 2 experiments in my file. My steps are described below:

In the file posted above (3rd post) in the main class I make  2 calls to the  MaspExample class()

test.CaseRun(120,heightField,widthField,speed,1024,10000,0.4,AODV,false,false,PRINTFILE);
test.CaseRun(120,heightField,widthField,speed,1024,10000,0.4,AODV,false,false,PRINTFILE);

This represents 2  experiments with the same attributes:  topology of 120 nodes, distributed on a rectangle  of widthField x heightField using packets of 1024 bytes with an application data rate of 10000bps
(0.4 is only used when I set my protocol, and the other flags control prints).


In order to ensure the same topology distribution in both experiments , I assigned a stream to the  RandomRectanglePositionAllocator in the file the following way. (Using stream 3)


      Ptr<UniformRandomVariable> x = CreateObject<UniformRandomVariable>();
      x->SetAttribute ("Min", DoubleValue (0));
      x->SetAttribute ("Max", DoubleValue (m_widthField));

      Ptr<UniformRandomVariable> y = CreateObject<UniformRandomVariable>();
      y->SetAttribute ("Min", DoubleValue (0));
      y->SetAttribute ("Max", DoubleValue (m_heightField));

      Ptr<RandomRectanglePositionAllocator> alloc =CreateObject<RandomRectanglePositionAllocator>();
      alloc->SetX (x);
      alloc->SetY (y);
      alloc->AssignStreams(3);


Now , this makes the topology have the same distribution every time but it does not affect the random variables inside  the AODV protocol. (mostly jitters I think)

I thought I could address this issue by using "AssignStreams" in the AodvHelper.  I have a node container called  clientNodes and a single mobile node called mobileNode
I tried setting this after the  InternetsStackHelper  like this:

  

  NodeContainer testContainer;
  testContainer.Add(clientNodes);
  testContainer.Add(mobileNode);
  aodvProtocol.AssignStreams(testContainer,4);


Or like this :

aodvProtocol.AssignStreams(clientNodes,4);
aodvProtocol.AssignStreams(mobileNode,4);


Both tries, still result in different packets received. I only made those simulations described above(test.caserun.......). Is it me doing something wrong or is it a bug in AODV?, are there any examples that shows AODV using AssignStreams to secure the same results in the experiments?

I just dont seem to find the reason after setting so many specific things
What does other people do in their experiments?? Just assume there will be different values of the same experiment and do an average for the results?

Best Regards,

Jack

[Praghur]

Hello all. 

Sorry if this is a very basic query. I am new to ns3 and nr module. 

I don't understand the purpose of 'AssignStream' in the NR module? What does it mean when the nrhelper assigns stream? I am referring to the following code from the example code 'cttc-nr-demo'. Can you please explain? I saw the explanation at Link but I am still not sure what 'stream' is. 

Thanks a lot in advance for any clarification. 

[Jack Higgins]

Hi Praghur, 

Please do not post questions on dead links, read existing post and if you cannot find the answer to your question is better to make a new post instead.

To answer your question. AssignStream is used to assign the stream value for the "random variables" used by the module.

More on random variables usage and reason in the official ns-3 documentation:

https://www.nsnam.org/docs/manual/html/random-variables.html

Regards,

Al.