What does the output of a simulation depend on?

[Chris Facchini]

Hi,

I've analyzed the output of a simulation that has been run on two
machines (differing for the architecture) and surprisingly it seems to
be different.
Before inspecting whether I have changed something in either one of
the installations, I'd like to pose a question just to be sure what's
happening.

My question is, does the output of a simulation depend only on the
seed or does it depend also on the machine architecture (eg i386 vs
amd64)?

Best,
Chris

[Mathieu Lacage]

if there are floating point calculations, yes, the result of these
operations will depend on architecture (i386 vs amd64), hence the
result of your simulation.

I would love to know if http://code.nsnam.org/mathieu/ns-3-time avoids
the difference across architectures for you.

Mathieu
--
Mathieu Lacage <mathieu...@gmail.com>

[Chris Facchini]

On Sep 1, 12:17 pm, Mathieu Lacage <mathieu.lac...@gmail.com> wrote:
> if there are floating point calculations, yes, the result of these
> operations will depend on architecture (i386 vs amd64), hence the
> result of your simulation.

Thanks for the clarification. OK, that's fair enough.

However, I have run the same code on two machines having the same
architecture[1] and the results still differ significantly.
Now, since the ns-3 version I use has been considerably extended, I'll
try to run some tests with a vanilla release and I'll keep you posted.

> I would love to know if http://code.nsnam.org/mathieu/ns-3-time avoids
> the difference across architectures for you.

I can give it a try and see if that solves the issue.

[1] Just for the records, both machines run i386 Linux. They differ in
terms of kernel version and GCC (2.6.27+4.4.4 vs 2.6.23+4.2.3,
respectively), among perhaps other things that I haven't thought of
yet.

[Chris Facchini]

OK, I think I got it.

On Sep 1, 12:17 pm, Mathieu Lacage <mathieu.lac...@gmail.com> wrote:
>
> if there are floating point calculations, yes, the result of these
> operations will depend on architecture (i386 vs amd64), hence the
> result of your simulation.

There's also another aspect involving calculation precision: the GNU
Scientific Library (GSL).
Using the script at http://www.nsnam.org/bugzilla/show_bug.cgi?id=730
[direct link: http://www.nsnam.org/bugzilla/attachment.cgi?id=842 ] I
was able to register:
- 35630 RxOk and 8 RxErr with GSL enabled
- 35517 RxOk and 365 RxErr with GSL disabled

Which translates in a FER about .0002 if GSL is enabled and .0102 if
GSL is disabled.
The difference, to me at least, is huge and I'm afraid it might affect
the reproducibility of results.

What do you think?

Chris

[Mathieu Lacage]

On Wed, Sep 1, 2010 at 15:09, Chris Facchini <c.fac...@gmail.com> wrote:

>  - 35630 RxOk and 8 RxErr with GSL enabled
>  - 35517 RxOk and 365 RxErr with GSL disabled
>
> Which translates in a FER about .0002 if GSL is enabled and .0102 if
> GSL is disabled.
> The difference, to me at least, is huge and I'm afraid it might affect
> the reproducibility of results.
>
> What do you think?

If you see a difference, it's because you are using
dsss-error-rate-model.cc and if so, you can see that it says:

NS_LOG_WARN ("Running a 802.11b CCK Matlab model less accurate than
GSL model");

To summarize, you have been warned :)

[Mathieu Lacage]

On Wed, Sep 1, 2010 at 15:09, Chris Facchini <c.fac...@gmail.com> wrote:

>  - 35630 RxOk and 8 RxErr with GSL enabled
>  - 35517 RxOk and 365 RxErr with GSL disabled
>
> Which translates in a FER about .0002 if GSL is enabled and .0102 if
> GSL is disabled.
> The difference, to me at least, is huge and I'm afraid it might affect
> the reproducibility of results.
>
> What do you think?

If you see a difference, it's because you are using

dsss-error-rate-model.cc and if so, you can see that it says:

NS_LOG_WARN ("Running a 802.11b CCK Matlab model less accurate than
GSL model");

To summarize, you have been warned :)

Mathieu
--
Mathieu Lacage <mathieu...@gmail.com>

[Chris Facchini]

On Sep 1, 4:29 pm, Mathieu Lacage <mathieu.lac...@gmail.com> wrote:
>
> NS_LOG_WARN ("Running a 802.11b CCK Matlab model less accurate than
> GSL model");
>
> To summarize, you have been warned :)

Oops! I totally missed it.
Thanks for the heads-up!

[PS: I'd love the 'NS_LOG_WARN' were a 'NS_LOG_UNCOND' but it'd
probably annoy all the other ns-3 users :]

[Gustavo Carneiro]

I think warnings should be shown by default.  But it should use a Python-style warning filter.  By default, Python warnings appear only once per (source file, line number) pair that emits the warning.  Thus, if there is warning inside a function that is called 1000 times, and the warning message is always the same and emitted from the same line, the warning is printed only the first time, and suppressed the remaining times.  With this kind of filter, I don't think anyone would object to warnings being printed by default.  The problem is finding time to  implement the filter...

Just my 2 cents.

 

--
You received this message because you are subscribed to the Google Groups "ns-3-users" group.
To post to this group, send email to ns-3-...@googlegroups.com.
To unsubscribe from this group, send email to ns-3-users+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/ns-3-users?hl=en.

--
Gustavo J. A. M. Carneiro
INESC Porto, UTM, WiN, http://win.inescporto.pt/gjc
"The universe is always one step beyond logic." -- Frank Herbert

[Mathieu Lacage]

On Wed, Sep 1, 2010 at 16:46, Chris Facchini <c.fac...@gmail.com> wrote:
>> NS_LOG_WARN ("Running a 802.11b CCK Matlab model less accurate than
>> GSL model");
>>
>> To summarize, you have been warned :)
>
> Oops! I totally missed it.
> Thanks for the heads-up!
>
> [PS: I'd love the 'NS_LOG_WARN' were a 'NS_LOG_UNCOND' but it'd
> probably annoy all the other ns-3 users :]

Fee free to file a bug and link to gustavoi's reply.