Master thesis about Talos noise finished

[Jan Larres]

Hi all,

I recently finished my Master's thesis about noise/variance in the Talos
tests. Some of you may remember that I gave a talk about my work at the
All Hands in September, so here now is the finished write-up:

http://majutsushi.net/stuff/thesis.pdf

Short summary of the important points:

- Memory randomization seems to have a significant impact on the
variance, other "external" factors like concurrently running,
unnecessary programs (like a full desktop environment) are also
measurable but not quite as pronounced.
- There is some variance in the number and order of events that are
used during the tests, but they don't seem to correlate with the
test results.
- I also have a suggestion for a statistical technique that should
give more reliable outlier detection than the t-test that is
currently being used by the graph server.

I hope it's somewhat useful. If there's anything unclear feel free to
ask!

Cheers,
Jan

[Chris AtLee]

Hi Jan,

Very interesting, thank you for sharing this.

I wrote the current system for detecting outliers, and am very eager to
give your suggestions a try.

Cheers,
Chris

[Johnathan Nightingale]

Yeah, this work is promising - filed https://bugzilla.mozilla.org/show_bug.cgi?id=706912

J

> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning

---
Johnathan Nightingale
Director of Firefox Engineering
joh...@mozilla.com