Going Faster: Mochitest browser chrome test results

[William Lachance]

Hey,

Just thought I'd post something about this before I head out for the
weekend. My latest mini-project for the GoFaster project
(https://wiki.mozilla.org/ReleaseEngineering/BuildFaster) has been to
profile the mochitest browser chrome tests. Jonathan Griffin sent me a
link to some existing timings we had for these tests in elasticsearch
and, similar to what I did with XPCShell
(https://bugzilla.mozilla.org/show_bug.cgi?id=617503), went about
writing a script to parse these JSON results (see attached csv file with
results as well as the python script I used to generate them). The
results are printed in seconds, because I find that measure more
comprehensible.

Between other things, I've been staring at these results over the last
few days and, here's my major findings:

1. Windows seems to be quite a bit slower than Linux. I'm guessing this
is probably disk i/o related (relevant bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=675363), but have only
confirmed that I'm not seeing this problem on Windows VM (where the
results are quite comparable with its Linux host). Sort of waiting to
see what we do with the aforementioned bug.
2. Mac also seems quite slow for some unknown reason.
3. 32-bit Linux seems slower than 64-bit Linux? Very strange.
4. Some of the tests seem to have absurdly low running times (0 in some
cases).

Feel free to reply with your pet comments and theories!

Will

[William Lachance]

For some reason, those attachments didn't go through. I posted them here:

http://people.mozilla.com/~wlachance/parse-mochichrome.py
http://people.mozilla.com/~wlachance/mochichrome-results.csv

Will

[William Lachance]

On 08/05/2011 08:30 PM, William Lachance wrote:

> Hey,
>
> Just thought I'd post something about this before I head out for the
> weekend. My latest mini-project for the GoFaster project
> (https://wiki.mozilla.org/ReleaseEngineering/BuildFaster) has been to
> profile the mochitest browser chrome tests. Jonathan Griffin sent me a
> link to some existing timings we had for these tests in elasticsearch
> and, similar to what I did with XPCShell
> (https://bugzilla.mozilla.org/show_bug.cgi?id=617503), went about
> writing a script to parse these JSON results (see attached csv file with
> results as well as the python script I used to generate them). The
> results are printed in seconds, because I find that measure more
> comprehensible.
>
> Between other things, I've been staring at these results over the last
> few days and, here's my major findings:
>

> ...

So jgriffin pointed out to me that there are different variations within
certain builds (debug and opt), which my old script was conflating.
After accounting for that issue, we see that the times for non-debug
builds are roughly the same across platforms (with Windows somewhat
slower, probably due to disk issues).

win64-opt: 808.515
macosx-debug: 2743.529
macosx-opt: 690.3
linux64-opt: 598.066
linux-debug: 2232.424
linux-opt: 633.295

So there may or may not be questions of why debug builds are so much
slower (or at least I have that question), but it seems that the
performance between platforms is at least comparable.

Will