xpcshell debug on Win7 take long on the minis
armenzg
unit tests from the builders to the minis.
The last standing platform was Win2003 for debug unit tests which
would move to Win7 debug unit tests.
2 out of the 3 perma-oranges have been fixed today.
philor pointed out that xpcshell takes quite long on Windows 7 on the
minis compared to the runs on the IX/VMs combination for xpcshell.
I am writing this so we can all be aware of it when we bring this up
on Tuesday. For a small summary keep on reading or you can jump to the
full post with all the data and charts.
See you on Tuesday,
Armen
[1] http://armenzg.blogspot.com/2010/12/xpcshell-case-on-windows.html
PS = My apologies if my I used the averages for only one day as it was
the easiest data I could collect.
###############
The numbers:
* xpcshell debug on Win2003 (average in mins) - 46 mins
* xpcshell debug on Win7 (average in mins) - 106 mins
It would have not been as bad if it didn't become the new longest test
suite to run but it has become.
The previous longest pole was:
* xpcshell opt on Win7 x64 (average in mins) - 80 mins
On my blog post I mention that the world would be happier if we
managed to run xpcshell in less time or break it into pieces. If
xpcshell on windows did not take that long, Window's longest poll
would instead be "mochitest-other" or "dromaeo" which would take
around 60 mins which is comparable to "mochitest-4 debug for
Fedora" (72 mins).
Ted Mielczarek
> The numbers:
> * xpcshell debug on Win2003 (average in mins) - 46 mins
> * xpcshell debug on Win7 (average in mins) - 106 mins
>
> It would have not been as bad if it didn't become the new longest test
> suite to run but it has become.
> The previous longest pole was:
> * xpcshell opt on Win7 x64 (average in mins) - 80 mins
It sounds to me like the problem is "xpcshell tests take a long time
to run on Windows 7", since even the opt builds take extremely long to
run. We should try tracking down the root cause here, this is not
reasonable. I suspect there's some bug or bad interaction with our
xpcshell tests on Windows 7 that's making them take entirely too long
to run. I don't think we should look at splitting up the suite or
anything until we've determined why it's taking so long.
I happen to know that there are a lot of developers with Windows 7
machines nowadays, so a good first step would be to have some people
download builds+tests and try running the same commands we run on the
test slaves locally and see how long they take.
-Ted
armenzg
> It sounds to me like the problem is "xpcshell tests take a long time
> to run on Windows 7", since even the opt builds take extremely long to
> run
>
Not just the hardware but maybe the OS (win7) might be at fault.
http://armenzg.blogspot.com/2010/12/test-suites-on-windows.html
xpcshell and reftests run worst. All other test suites are unaffected
by the switch (neither hardware nor OS nor graphics bound).
I will post the steps that we run for xpcshell so people can try to
reproduce/debug.
Cheers,
Armen
armenzg
#!/bin/sh.exe
FTP="http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds"
BRANCH=mozilla-central
OS=win32
TYPE=debug
BINARY=firefox-4.0b8pre.en-US.win32.zip
TESTS_PACKAGE=firefox-4.0b8pre.en-US.win32.tests.zip
if [[ ("$#" != 1 || ! ($1 =~ ^[0-9]+$) ) ]]; then
echo "One argument required in the form of a unix timestamp"
echo "We want to build an ftp tinderbox-builds URL:"
echo "$FTP/$BRANCH-$OS-$TYPE/\$UNIX_TIMESTAMP"
exit 1
fi
UNIX_TIMESTAMP=$1
wget --progress=dot:mega -N $FTP/$BRANCH-$OS-$TYPE/$UNIX_TIMESTAMP/
$BINARY
unzip -o $BINARY
wget --progress=dot:mega -N $FTP/$BRANCH-$OS-$TYPE/$UNIX_TIMESTAMP/
$TESTS_PACKAGE
unzip -o $TESTS_PACKAGE bin* certs* xpcshell*
if [ ! -d firefox/plugins ]; then mkdir firefox/plugins; fi && cp bin/
xpcshell.exe firefox && cp -R bin/components/* firefox/components/ &&
cp -R bin/plugins/* firefox/plugins/ && python -u xpcshell/
runxpcshelltests.py --symbols-path=symbols --manifest=xpcshell/tests/
all-test-dirs.list firefox/xpcshell.exe
Ehsan Akhgari
> On Dec 4, 10:28 am, Ted Mielczarek <t...@mielczarek.org> wrote:
>> It sounds to me like the problem is "xpcshell tests take a long time
>> to run on Windows 7", since even the opt builds take extremely long to
>> run
>>
> Guess what? You are right.
>
> Not just the hardware but maybe the OS (win7) might be at fault.
> http://armenzg.blogspot.com/2010/12/test-suites-on-windows.html
> xpcshell and reftests run worst. All other test suites are unaffected
> by the switch (neither hardware nor OS nor graphics bound).
Is it possible that the difference would be in our startup/shutdown
times? I noticed that we're about 7% slower in our startup time on
Windows 7 compared to Windows XP <http://is.gd/in0q9>, and about 50%
slower in our shutdown times on the same OSes <http://is.gd/in0I9>.
(Note that the single most unusual thing about the xpcshell tests is
that we basically startup and shutdown Gecko for every test, compared to
once per suite).
If this is really the case, we should probably have blocking bugs on
file for investigating the issue...
Cheers,
Ehsan
Clint Talbert
> Is it possible that the difference would be in our startup/shutdown
> times? I noticed that we're about 7% slower in our startup time on
> Windows 7 compared to Windows XP<http://is.gd/in0q9>, and about 50%
> slower in our shutdown times on the same OSes<http://is.gd/in0I9>.
>
> (Note that the single most unusual thing about the xpcshell tests is
> that we basically startup and shutdown Gecko for every test, compared to
> once per suite).
>
> If this is really the case, we should probably have blocking bugs on
> file for investigating the issue...
Ehsan, you're absolutely right. I filed bug 617503 [1], on this
yesterday and started trying to profile it. There is ongoing analysis
in that bug (many many thanks to Jimm), but yes, we're spending all our
time in startup/shutdown.
Clint
[1]: https://bugzilla.mozilla.org/show_bug.cgi?id=617503