****How to repeat simulations with the same results*****

[shahin]

Hi,

If anyone can help me in this matter I really appreciate it.

I have read the OMNeT++ manual and I need to have multiple runs of my simulations with different seeds. Hence each run represents a different result compared to the one before. Everything is alight until this point for me. 

lets say X = {x1, x2, x3, x4, x5} and Y = {y1, y2, y3, y4, y5} where X holds the results for my first batch of simulation runs (running 5 times different seeds) and Y holds the results for my second batch of runs (also 5 times with different seeds). X and Y are under the exact same simulation condition. I need X = Y. Which means that if I run my simulation batch even 100 times (X, Y, Z, W, M and...), the results for all of the 100s of them be the same. I just needed to make sure x1, x2, x3, x4, x5 are not equal to each other (same for y1, y2, y3 and ...) meaning that within each batch of simulation, results are different so I can average over them and come up with a statistically reliable results.

When I put this line in my .ini file: "seed-set = ${runnumber}" or this "seed-set = ${repetition}" I end up with different results among my X and Y set (for example). Meaning that I repeat the simulation 5 times (different seed), and then another 5 times (different seed) and compare one of the values from the first batch of repeat with second batch (for example end-to-end delay) and see that they are not the same.

There is another option explained in the manual "seed-set = ${5,6,8..11}", which when I use it, I end up with this error "<!> Error during startup: Scenario generator: no such variable: ${-1--1-0}

"

So the question is, how can I set the seed so that while in a "repeat = 5" each of 5 repeats have different results but if I repeat this process (runs with 5 repeats different seeds again and again), I come up with exact same results as the first batch of 5 repeats.

Pleaaaase help... Really need this.

Regards,

[Leonardo Maccari]

On 13/04/2012 13:21, shahin wrote:
> Hi,

>
> There is another option explained in the manual "seed-set =
> ${5,6,8..11}", which when I use it, I end up with this error "<!> Error
> during startup: Scenario generator: no such variable: ${-1--1-0}
> "

you may be missing the "! repetition" keyword:

http://omnetpp.org/doc/omnetpp/manual/usman.html#sec946

When I need something like that I use a script as the following:

==============================
rm rng.ini
echo "[General]" >> rng.ini
echo "num-rngs = 9" >> rng.ini

RUNS=30
LINE="$RANDOM"

for i in `seq 1 $RUNS`; do
LINE=$LINE", $RANDOM";
done;
echo "seed-1-mt=\${$LINE! repetition}" >> rng.ini
echo "seed-2-mt=\${$LINE! repetition}" >> rng.ini
echo "seed-3-mt=\${$LINE! repetition}" >> rng.ini
echo "seed-4-mt=\${$LINE! repetition}" >> rng.ini
echo "seed-5-mt=\${$LINE! repetition}" >> rng.ini
echo "seed-6-mt=\${$LINE! repetition}" >> rng.ini
echo "seed-7-mt=\${$LINE! repetition}" >> rng.ini
echo "seed-8-mt=\${$LINE! repetition}" >> rng.ini
echo "seed-0-mt=\${$LINE! repetition}" >> rng.ini
==============================

and then include rng.ini

--
Leonardo Maccari, Post-doc researcher@University of Trento
Tel: +39 0461 285323, project website: www.pervacy.eu

[Shahin Shariat]

Thanks for your answer. I added the ! repetition to the end of the bracket but still the results are not repeatable. 

Lets make the problem simpler. What do I need to add into my .ini file to force the whole simulation to use one random number generator with the same seed so my results are repeatable? Currently I cannot generate any repeatable results.

Can you please help me with this first?

Regards,

Shahin

--
Sent from the OMNeT++ mailing list. To configure your membership,
visit http://groups.google.com/group/omnetpp

[Alfonso Ariza Quintana]

If the code don’t have errors, the simulation, with the same seed and with the same simulation parameter will have the same results.

[najam sama]

hi
i face a problem to create ring topology please help me

[Shahin Shariat]

Dear Alfonso,

I haven't even changed a single code. Everything is INETMANET and OMNET itself. I recently downloaded OMNET++ 4.2.2. Could it be this causing the problem? 

In my .ini file, I have this line:

seed-0-mt = 5

But I am getting different results between every run that I make. 2 month ago i used to run my simulation with this configuration and I was getting the same results. I don't know what has happened suddenly!!

Alfonso if there is any way I can fix this please help...

Regards,

Shahin.

[Alfonso Ariza Quintana]

what version of inetmanet
inetmanet-2.0 or inetmanet

---

Date: Sun, 15 Apr 2012 13:02:55 +0100

Subject: Re: [Omnetpp-l] ****How to repeat simulations with the same results*****

From: sh.sh...@gmail.com
To: omn...@googlegroups.com

[Shahin Shariat]

Last week I downloaded the latest version by running this:

git://github.com/inetmanet/inetmanet.git  in the Linux Terminal.

the "Version" text file within the package says:    inet-1.99.3

Is this the wrong version?

Regards,

Shahin

[Shahin Shariat]

Just to make sure my ini file doesn't have any problem I tried using the example net80211_aodv of INETMANET. Using this example, since the mobility is stationary, all runs have same results. Then, I changed the mobility as below:

**.host*.mobilityType = "RandomWPMobility"

**.host*.mobility.speed = 3mps

**.host*.mobility.waitTime = 5s

Running the simulation again and again resulted different AODV_totalsent and totalReceived (count) values. I haven't changed any value of this example and everything is the default.

Alfonso can you do the same and see if you get different results?

Regards,

Shahin.

[Leonardo Maccari]

On 16/04/2012 13:55, Shahin Shariat wrote:

if you include the random seeds from a file (as with the script I cut in
a previous email) in the resulting .vec file the seeds are reported
(the iterationvars value), check if the seeds are the same for each run
you're testing.
if they are the same and still have different results something doesn't
work properly.

ciao,
leonardo.

[Shahin Shariat]

Thanks for your reply. I haven't ever ran my simulation with scripts. Can you explain where should i put the script that you mentioned? If you explain it little more clear I really appreciate it.

Regards,
Shahin

[shahin]

Dear Leonardo,

Ok i tried running my simulation with the script you provided. I have attached the the two screenshots from the results after each set of my runs. As you can see they are completely different. I don't really know what else to do. The results are not repeatable!! 

[Leonardo Maccari]

On 17/04/2012 22:48, shahin wrote:
> <https://lh4.googleusercontent.com/-7fIFEhMTZ5o/T43WvwefolI/AAAAAAAAAAY/g2TpGUd5FRo/s1600/Screenshot-1.jpg>
>
> <https://lh3.googleusercontent.com/-S3meiFbp_50/T43WqxDCG_I/AAAAAAAAAAM/qLOqboj_MyA/s1600/Screenshot.jpg>

>
>
> Dear Leonardo,
>
> Ok i tried running my simulation with the script you provided. I have
> attached the the two screenshots from the results after each set of my
> runs. As you can see they are completely different. I don't really know
> what else to do. The results are not repeatable!!

did you check in the result file if the seeds are the same for both the
simulations?

[Alfonso Ariza Quintana]

I have actualized the code and replace the list container for STL container, this should resolve the problem.
apparently the list used by AODVUU doesn't guarantee that the order of the elements stored in the list is the same in all the simulations

---

Date: Tue, 17 Apr 2012 13:48:01 -0700
From: sh.sh...@gmail.com

To: omn...@googlegroups.com
Subject: Re: [Omnetpp-l] ****How to repeat simulations with the same results*****

[Jai]

Dear 
         Alfonso Ariza Quintana ,

Sir Please help me. How could I set Up Network simulation to test parameter given in the below title.

"COMPUTER NETWORKS AND THEIR PERFORMANCE EVALUATION ON THE BASIS OF COMMUNICATION SPEED, SECURITY RISKS, NETWORK CONTROLLING AND SCALABILITY"

Using wired network. Please help me.

[Shahin Shariat]

Pleaseee make a new thread for your new questions... When you see in a thread people are talking about something other than what your question is, take the hint and please don't ask questions unrelated to that topic.

[shahin]

Dear Alfonso,

So should I get the new version of AODV from "https://github.com/aarizaq/inetmanet-2.0" ? Coz at the moment I am using the INETMANET and not the INETMANET 2.0...

What do you suggest that I do?

Regards,

Shahin

[Alfonso Ariza Quintana]

 the latest code is in inetmanet-2.0, but if you want to use inetmanet you can include the modifications.

---

Date: Wed, 18 Apr 2012 03:49:34 -0700

[shahin]

Dear Alfonso,

Are you sure the new AODV code in INETMANET 2.0 resolves the issue?

I just tried it and I'm still getting different results with the same seed. Can you please confirm this?

regards,

Shahin

[Alfonso Ariza Quintana]

I have removed the old timer container, I can't find the cause of the problem

---

Date: Sun, 29 Apr 2012 10:32:00 -0700

[N.omar]

I am facing the same problem !!!! I am using Omnet4.1 and inetmanet ,,, what i should do ?! 

[Alfonso Ariza Quintana]

inetmanet-2.0 doesn't work with omnet 4.1 it needs at least version 4.2

---

Date: Wed, 5 Jun 2013 02:42:07 -0700
From: saher.l...@gmail.com
To: omn...@googlegroups.com
CC: aari...@hotmail.com

--

--
Sent from the OMNeT++ mailing list. To configure your membership,
visit http://groups.google.com/group/omnetpp
 

---
You received this message because you are subscribed to the Google Groups "omnetpp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to omnetpp+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[N.omar]

I have installed Omnet4.3 and inetmanet2.0 

but I get the following error In the console 

"<!> Warning: opp_run: Cannot check library ../../../src/inet: The specified module could not be found

<!> Error during startup: Cannot load library '../../../src//libinet.dll': The operation completed successfully. " 

[Shahin Shariat]

Have you built INETMANET 2.0 completely? What is the output when you build it?

You received this message because you are subscribed to a topic in the Google Groups "omnetpp" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/omnetpp/DnFPPvlvRXk/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to omnetpp+u...@googlegroups.com.

[N.omar]

I solved the error of inetmanet2.0  by update the date of RSVPPacket.msg file

Now I faced another problem with TCP I note that inetmanet 2.0 contains an warnings in each example about tcp 

for example it put a warning/error on TcpAlgorithmType in the .ini file(Unused entry doesn't match any parameter) !!!  and  another parameters  about TCP 

  so what i should do to solve the problem of inetmanet  with Tcp ?!