Performance and stability tests with extensions installed.
John J. Barton
article comparing IE8, Safari, Chromium, and Firefox 3.1. The key
advantage of Firefox was add-ons; a non-advantage was speed. The article
makes clear what we already know: users that like and use Firefox
install add-ons. Consequently, performance and stability tests on stock
Firefox are not important. What counts is performance and stability of
Firefox + add-ons. Are these kinds of tests run during development?
jjb
Ted Mielczarek
extensions installed. The obvious question here would be "what
extensions"? I think a possible avenue here would be to survey some
top extensions, and find out *why* they make Firefox slow. It may be
that we can fix the underlying problem, or figure out a better way to
do what the extension is trying to do. Perhaps we can figure out some
set of things that extensions are likely to do, and have documentation
available for extension authors to let them know of possible
downsides, and ways to avoid them.
-Ted
Mike Shaver
<ted.mie...@gmail.com> wrote:
> I can tell you that no, none of our automated tests run with
> extensions installed.
We have the ability to run automated leak tests now with the top 2000
extensions, and I believe that Tomcat does so periodically (256
extensions at a time or something).
Mike
Ted Mielczarek
> On Thu, Mar 26, 2009 at 1:26 PM, Ted Mielczarek
> extensions, and I believe that Tomcat does so periodically (256
> extensions at a time or something).
Ok, I guess I should have said "none of the tests that run on
Tinderbox", which are the sort of things that developers monitor on a
regular basis.
-Ted
L. David Baron
I don't know of any, but I definitely agree we should be running
such tests. (Note that differences between performance, memory use,
or stability between with and without an extension or combination of
extensions could be a sign of bugs either in those extensions or in
Firefox itself.) It would probably be even better to publish such
tests results on a.m.o so users know how the extension affects
performance (no matter whose fault it is; if it's actually a Firefox
bug that would put more pressure on getting it filed and fixed).
(However, a big issue with testing extensions is that
you often need to take some user action to turn the extension on.)
I worry a bit that we publish extensions on a.m.o whose end-user
licenses explicitly forbid publication of test results, though.
-David
--
L. David Baron http://dbaron.org/
Mozilla Corporation http://www.mozilla.com/
John J Barton
> On Thursday 2009-03-26 08:15 -0700, John J. Barton wrote:
>> The March 26th edition of the New York Times Business section has an
>> article comparing IE8, Safari, Chromium, and Firefox 3.1. The key
>> advantage of Firefox was add-ons; a non-advantage was speed. The article
>> makes clear what we already know: users that like and use Firefox
>> install add-ons. Consequently, performance and stability tests on stock
>> Firefox are not important. What counts is performance and stability of
>> Firefox + add-ons. Are these kinds of tests run during development?
>
> I don't know of any, but I definitely agree we should be running
> such tests. (Note that differences between performance, memory use,
> or stability between with and without an extension or combination of
> extensions could be a sign of bugs either in those extensions or in
> Firefox itself.) It would probably be even better to publish such
> tests results on a.m.o so users know how the extension affects
> performance (no matter whose fault it is; if it's actually a Firefox
> bug that would put more pressure on getting it filed and fixed).
In addition these tests would help guide work on performance. If
extension tend to cause problems in a certain area, say lots of event
handlers on tab switching, then you'll want to invest in that area of
tab switching rather than other areas. Since the user will experience
the combination package of firefox and addons, in a way profiling only
firefox is a kind of 'synthetic' test that tells you some useful
information but is not the real deal.
>
> (However, a big issue with testing extensions is that
> you often need to take some user action to turn the extension on.)
Yes, but we'd be interested in just the 'incidental' overhead of
extensions (startup, tab switching, page loading, places?), not their
own performance, so the actions on the extension would not be very
extensive.
>
> I worry a bit that we publish extensions on a.m.o whose end-user
> licenses explicitly forbid publication of test results, though.
Maybe amo can have an opt-in field and we can lobby the top 50
extensions. And maybe the actions to start the extension could be
somehow specified at that time.
jjb
Boris Zbarsky
>> (However, a big issue with testing extensions is that
>> you often need to take some user action to turn the extension on.)
>
> Yes, but we'd be interested in just the 'incidental' overhead of
> extensions (startup, tab switching, page loading, places?), not their
> own performance, so the actions on the extension would not be very
> extensive.
I think dbaron's point was that just installing the extension doesn't do
anything for some extensions, so just installing and running is a
meaningless test. For example, consider something like adblock with no
filter lists installed (the default install; lists are installed
separately).
-Boris
Philip Chee
> I worry a bit that we publish extensions on a.m.o whose end-user
> licenses explicitly forbid publication of test results, though.
AMO can have a TOS which says you automatically waive your EULA (to AMO)
if you submit your addon to us.
Phil
--
Philip Chee <phi...@aleytys.pc.my>, <phili...@gmail.com>
http://flashblock.mozdev.org/ http://xsidebar.mozdev.org
Guard us from the she-wolf and the wolf, and guard us from the thief,
oh Night, and so be good for us to pass.
Gervase Markham
> AMO can have a TOS which says you automatically waive your EULA (to AMO)
> if you submit your addon to us.
Or we could say that we won't host addons with such consumer-unfriendly
terms at all.
We have a fairly big carrot in the form of AMO; not using AMO makes
distributing your extension more difficult. I don't think it's
unreasonable to use that carrot to make the world a more
consumer-friendly place.
Gerv