Omnet and my PhD proposal

[Mohsen Sichani]

Dear friends

I am using Omnet for several months and I worked with it for several month. Now, I have my upcoming proposal defense, I have some questions and I will be so honored if any body can help me with the answers. I also appreciate any questions/comment that you think examiners may ask or be interested (about the simulation).

1st: I noticed that the number of simulation runs is less in Omnet in comparison to other simulators. I read some papers which were written by developers and they did five runs (Yes , with different seeds). It seems  it is low, but trustable, why is it trustable?

2nd: I have noticed that authors also mentioned Confidence interval in their research. Actually, I searched the forum and books, but I could not find any available link or very useful content. Based on what I read, I understood that we cannot define confidence interval in "ini" or "ned". We only can measure that after simulation as I saw it in "compute scalar" part in "dataset".So, how this confidence interval can help us to interpret the low number of simulation runs and how is it useful in interpreting results?

Any help is highly appreciated.

Regards

Mohsen

[Michael Kirsche]

Regarding the confidence interval, Wikipedia has a longer article on it: https://en.wikipedia.org/wiki/Confidence_interval

As for the number of simulation runs with different seeds, that totally depends on what you are simulating / wishing to examine / looking for.
If, for example, you want to analyze the performance of some routing protocol, then a multitude of runs with different set-ups is necessary. If you want to proof that your model behaves like a real-life protocol, you might not need many runs (might or might not). There is no definite rule of how many runs you would need to get the results that you are looking for. OMNeT can do as many runs as you like... it's up to you.

[Ivo Calado]

Mohsen,
    a good source about performance analysis is the book of Jain (http://www.amazon.com/The-Computer-Systems-Performance-Analysis/dp/0471503363/ref=pd_sim_b_1)

[]'s

--
You received this message because you are subscribed to the Google Groups "omnetpp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to omnetpp+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Ivo Calado

Quidquid latine dictum sit, altum viditur

Putt's Law:
       Technology is dominated by two types of people:
               Those who understand what they do not manage.
               Those who manage what they do not understand.

[Mohsen Sichani]

Ivo/Michael

Thanks a lot for your thoughfulness. I appreciate that. Ivo that is really a good book, thanks for that.

Michael

I have seen that Alfonso in his paper did 5 runs with different seeds for routings purpose. Thus, it seems five runs is acceptable, ,but some of my friends who work with other simulators , runs the simulation for more than 20 times for a scenario. Is there any specific reason for that or Is it just a myth?

Thanks once again

Cheers

Mohsen 

"For each evaluated scenario, five runs with

different seeds have been executed. The de-

picted figures represent the mean values of the

measured metrics as well as the 95% confi-

dence interval." from: Cooperative layer-2 based routing approach for hybrid wireless mesh networks 

--
You received this message because you are subscribed to a topic in the Google Groups "omnetpp" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/omnetpp/7KDaUjwZJj0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to omnetpp+u...@googlegroups.com.

[Alfonso Ariza Quintana]

There are several aspect that it is necessary to consider

Time necessary to run the simulation

Simulated time

Variance of the results

 

The first is a practical problem and is derived from the second and there is a relation with the third.

 

If you simulated long time, the results usually converge (if the simulation is stable), in this case the Variance is small, and if the variance is small you don’t need a big number of simulations.

In other hand, if the simulated time is big, the time necessary to run the simulation can be really enormous.

 

In my works, I have replace a big number of simulations for longest simulations with several (sometime 100) millions of frames.

In lot of papers the simulated time is in the interval of 300-900 and the number of frames is relatively small, but my minimum time is 3000 seconds. If you simulate a small time, the results usually have a big variance (probably you are measuring the transitory) but if the simulation is stable and you simulate a long time with millions of frames, the variance usually is small. In this case, when you repeat the simulation several times and you find that the variance is small, you can use a small set of different seeds.

[Mohsen Sichani]

Thanks dear Alfonso.

I greatly appreciate that, This was really helpful and now I have a standard in my mind for my future work.

Thanks once again.

Cheers

Mohsen

[Ivo Calado]

Mohsen,

    I usually use the attached script to calculate the number of required samples. As Alfonso said you may obtain such samples from different simulations or just splitting a large simulation in minor frames.

Depending of variance of your samples you may need a large number of samples in order to achieve a confidence level of 95%. For instance, the files res.medias and res2.medias represent two set of samples. After run the script on the first file (./calculateSample.sh res) you'll see that due to high variability you'll need to run several times. On the other hand in the second one, such requirement is not necessary.

My 2 cents...

[Mohsen Sichani]

Ivo

Thanks a lot. It should be really helpful. I greatly appreciate that. I will study them today and I will get back to you. 

Cheers

Mohsen

[Alfonso Ariza Quintana]

 

I forgot comment something before,

 

There is a practical reason of to use a minimum of 10 seeds, with this number you can use the exponential confidence interval instead of the T-S.

T-S and exponential converge if the number of series is bigger or equal of 10.

[Mohsen Sichani]

Thanks Alfonso

I greatly appreciate that. I need to google lots of things to fully understand everythig that you /Ivo/Michael told me ;-), It is reallt amazing and I appreciate that;-)

Thanks a million

Mohsen

[Sebastian Šubik]

Dear Mohsen,

I just want to extent the excellent answers written so far with another question for you: Try to understand the reason for the spread of your results within your simulation model. It is important to understand, why your results are varying. 

Example: If you look at a simple Aloha model, you can simulate it as an analytical model and just solve the equitation. You will get a result of 18,2 % (I haven't looked it up right now) with no variance. Another way to get the result is to use the build in model from OMNET, but then you won't get the exact value. The results will spread over the different seeds, because you have a difference between the analytical model (unlimited no of sender, collision with your self, not depending on the simulated time) and the simulation model (limited no of sender, no self-collision, depending on the processed time/number of simulated events). 

So the source of the variance are the differences between the two models. Now you can use statistical analysis to describe and measure this difference and to optimize your results. You can either increase the no of simulated sender, or you can increase the number of simulation runs, etc... Just play around with some of the parameter and compare the statistical results (e.g. Confidence interval or variance). You can try to modify OMNET's aloha example to get better results as an exercise, with the standard model you won't reach 18,2% (or whatever the value from literature is...).

The advantage of the analysis of aloha is that you have the analytical model and you know the exact result. For your own work I assume you not not have such a luxury and you have to deal only with your simulation results.

For additional readings, my favorite book is http://www.amazon.com/Applied-Statistics-Probability-Engineers-Montgomery/dp/0470053046/ref=sr_1_3?s=books&ie=UTF8&qid=1415734985&sr=1-3&keywords=Statistic+engineers

You now should have a lot of things to read for the next days, if you have additional questions, please ask ;-)

Best Regards

Sebastian

[Mohsen Sichani]

Hi all of my friends Alfonso/Ivo/Sebastian

Alfonso

I greatly appreciate that. You provided me a good detailed answer. Now, I think I have a good understanding of the simulation models and results. Your answer was promising. 

Just one simple question 

Does T-S referes to "Takagi-Sugeno" as you mentioned in "T-S and exponential converge if the number of series is bigger or equal of 10."?

Thanks in advanced.

Ivo

It is a great piece of code, It makes my life easier. I appreciate that. Now, I can do a good performance evaluation. Special thanks for the book. It is a great book.

Sebastian

I appreciete your additional explanation which helped me to understand better the problem and solution. Many thanks for the book. It is a great book and I can refer to it and get additional information. I admire your thoughfulness.

Have a good day.

Cheers

Mohsen

[Alfonso Ariza Quintana]

Test student, it used to compute the confidence interval

[FEIT]

Hello Alfonso, would you please say what confidence interval value is used in INET showcase>>wireless>>sensornetwork>>powerConsumption.anf file? Is it 95% confidence interval by default? Thank you.