Recommended Addon performance testing
k m
for performance regression and compatibility with vanilla Firefox? The
most common cause of Firefox slowdown is a badly-coded plugin, addon
or inter-addon compatibility.
I know it is very difficult to test all addons for performance or
compatibility, but having the recommended addons tested separately or
together with other addons will help the user avoid installing faulty
addons.
Also commonly installed plugins like flash, acrobat, real player,
foxit, WMP, etc can be tested likewise and warn the user.
Some recommended addons are updated quite frequently so testing them
whenever a major Firefox update is due might be easier.
On a completely unrelated note, implementing auto-suggest in
addons.mozilla.org would be very helpful.
Alexander Limi
interest (at least from some of us here at Mozilla, not that I can
personally implement this, unfortunately ;) in making this happen.
We currently test with what we call "dirty profiles" (meaning profiles with
lots of history, bookmarks, etc) to discover performance regressions if they
happen, we should definitely do the same with at least the top 20 add-ons.
--
Alexander Limi · Firefox User Experience Team · http://limi.net
2010/2/24 k m <121...@gmail.com>
> _______________________________________________
> dev-apps-firefox mailing list
> dev-apps...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-apps-firefox
>
k m
> It has been raised several times in the past, and there's considerable
> interest (at least from some of us here at Mozilla, not that I can
> personally implement this, unfortunately ;) in making this happen.
>
> We currently test with what we call "dirty profiles" (meaning profiles with
> lots of history, bookmarks, etc) to discover performance regressions if they
> happen, we should definitely do the same with at least the top 20 add-ons.
>
> --
> Alexander Limi · Firefox User Experience Team ·http://limi.net
Thats quite unfortunate. Is there any reason why this cant be
implemented?
And are there any tools for a non-technical person to measure browser
performance (excluding 3dmark, futuremark, peacekeeper). Having 20-30
addons makes it a bit difficult identify problems with a standard
diagnostic. Even then crashes or incompatibilities can be identified
if they are annoying enough, its only those small lags or temporary
slowdowns which irritates me.
Alexander Limi
> Thats quite unfortunate. Is there any reason why this cant be
> implemented?
>
Read that again. ;)
I just said that I personally can't implement it, since I'm a UX person, not
a test setup person. :)
> And are there any tools for a non-technical person to measure browser
> performance (excluding 3dmark, futuremark, peacekeeper). Having 20-30
> addons makes it a bit difficult identify problems with a standard
> diagnostic. Even then crashes or incompatibilities can be identified
> if they are annoying enough, its only those small lags or temporary
> slowdowns which irritates me.
>
I have proposed something similar as a project for Mozilla Labs a week or
two ago, we'll see if they are interested in implementing it.
k m
No No, I just wanted to know if there was any technical reason for
Mozilla not to implement it. Lets hope Mozilla Labs develop something.
Thanks for the reply!
Shawn Wilsher
> No No, I just wanted to know if there was any technical reason for
> Mozilla not to implement it. Lets hope Mozilla Labs develop something.
Cheers,
Shawn
Clint Talbert
> Apart from checking addons for security, are recommended addons tested
> for performance regression and compatibility with vanilla Firefox? The
> most common cause of Firefox slowdown is a badly-coded plugin, addon
> or inter-addon compatibility.
pretty straightforward problem to solve, and I'd welcome any volunteers
who want to try their hand at it, I'm ctalbert on IRC.
One of the reasons we haven't invested the time to do this on a regular
basis is that we did do a preliminary study to see what the performance
impact of the top 100 add-ons were. We did this back last July, and we
found that out of a 100 add-ons only one had a significant performance
impact. A bug was filed on that specific add-on but I can't remember
what its name was.
Here's what we did:
* Install an add-on off the list
* Boot firefox, browse to a set of the top 100 Alexa sites (obtained
from a local server to eliminate network impacts)
* Compared the total time it took to go through the list with a baseline
run of firefox without extensions.
Since most of the add-ons didn't affect performance we decided that
doing this on an ongoing basis wasn't a huge priority, though it would
be interesting to try to quantify what happens when multiple extensions
are installed and interacting with each other. Also, I think that one
of the reasons we didn't see much of an impact is that the automation we
used here was quite basic and didn't actually interact with the add-on.
I think if we're really going to test this and get "real world" perf
numbers we need to mock up some interaction tests for these add ons or
at least set their preferences so that they are in an "active" state
while we do the performance test. Of course, everything you do in an
interactive test with the addon has the potential to impact the numbers
you create, so you can see where it becomes hard to compare apples to
apples.
If there are folks out there interested in helping to work on this, let
me know and we'd love to help you get started.
Clint
k m
As you said the problem is not with the network performance of Firefox
very few addons interfere with the network. I can in a roundabout
manner identify network problems through online benchmarks like
sunspider, peacekeeper etc. If the score is better with a clean new
profile then its probably the addons which reduces the performance.
The irritants are slow startup of the program, sluggish Menus, buttons
not responding immediately, taking too much time to shutdown etc. I
thought Mozilla used some kind of a tool to find out regressions,
that's the reason why I posted about this problem. Is there a way an
average user can monitor his App performance?
Asa Dotzler
> The irritants are slow startup of the program, sluggish Menus, buttons
> not responding immediately, taking too much time to shutdown etc. I
> thought Mozilla used some kind of a tool to find out regressions,
> that's the reason why I posted about this problem. Is there a way an
> average user can monitor his App performance?
Mozilla does empoy a set of tests across a set of machines that measure
and report start-up time, new window time, pageload time, etc.
See http://graphs.mozilla.org/dashboard/?tree=Firefox for example.
- A
k m
Yes, but can we measure performance through same or maybe a similar
tool? Maybe if we can, we could post numbers for specific addons in
AMO.
Alexander Limi
> Yes, but can we measure performance through same or maybe a similar
> tool? Maybe if we can, we could post numbers for specific addons in
> AMO.
>
As mentioned earlier, I have proposed a project for Labs to build a tool
that end-users can use to pin-point these slow-downs, especially since (as
Clint points out), most of the badly behaving extensions aren't actually on
AMO, but rather binary extensions and things like virus scanners and other
(IMO ;) crap. This would allow the community to find these things, and also
experiment with combinations of extensions, if there are combinatorial side
effects.
Then we could use this data to reach out and help people write better
extensions.
Clint Talbert
> On Fri, Feb 26, 2010 at 1:49 AM, k m<121...@gmail.com> wrote:
>
>> Yes, but can we measure performance through same or maybe a similar
>> tool? Maybe if we can, we could post numbers for specific addons in
>> AMO.
>>
>
> As mentioned earlier, I have proposed a project for Labs to build a tool
> that end-users can use to pin-point these slow-downs, especially since (as
> Clint points out), most of the badly behaving extensions aren't actually on
> AMO, but rather binary extensions and things like virus scanners and other
> (IMO ;) crap. This would allow the community to find these things, and also
> experiment with combinations of extensions, if there are combinatorial side
> effects.
moment, Mozmill and Mochitest come to mind. But none of that stuff was
written with a performance impact in mind, so you'd pay some very
non-zero overhead for using that tool to measure performance. I'd have
to think about that a bit more and do some experiments to see what
impacts there are from the measurement itself.
> Then we could use this data to reach out and help people write better
> extensions.
>
That'd obviously be a number one goal. The other number one goal would
be to identify a set of APIs that are somewhat common to extensions
(should a common set of APIs actually exist) that we could optimize.
Clint
Alexander Limi
> On 2/26/10 1:05 PM, Alexander Limi wrote:
>
>> As mentioned earlier, I have proposed a project for Labs to build a tool
>> that end-users can use to pin-point these slow-downs, especially since (as
>> Clint points out), most of the badly behaving extensions aren't actually
>> on
>> AMO, but rather binary extensions and things like virus scanners and other
>> (IMO ;) crap. This would allow the community to find these things, and
>> also
>> experiment with combinations of extensions, if there are combinatorial
>> side
>> effects.
>>
> I am thinking we have pieces of this tool existing in other tools at the
> moment, Mozmill and Mochitest come to mind. But none of that stuff was
> written with a performance impact in mind, so you'd pay some very non-zero
> overhead for using that tool to measure performance. I'd have to think
> about that a bit more and do some experiments to see what impacts there are
> from the measurement itself.
Right — it wouldn't be a tool that would be running normally, more like a
benchmark suite. So performance overhead would be fine.
> Then we could use this data to reach out and help people write better
>> extensions.
>>
>> That'd obviously be a number one goal. The other number one goal would be
> to identify a set of APIs that are somewhat common to extensions (should a
> common set of APIs actually exist) that we could optimize.
>
Right — hopefully there is some low-hanging fruit that we can fix, but even
knowing which extensions are the slow ones so we can make people aware of it
would be great. People might e.g. find it acceptable that a virus scanner
makes their browser 20% slower on their setup, but at least have the
information needed to make the decision on whether to install it or not.
k m
> Right — hopefully there is some low-hanging fruit that we can fix, but even
> knowing which extensions are the slow ones so we can make people aware of it
> would be great. People might e.g. find it acceptable that a virus scanner
> makes their browser 20% slower on their setup, but at least have the
> information needed to make the decision on whether to install it or not.
>
> --
> Alexander Limi · Firefox User Experience Team ·http://limi.net
Most users readily blame Firefox for a slow addon, so if you guys can
put out an advisory or some sort of a performance rating system in
AMO, the user can install an addon at their own discretion. Some users
can stand a slight loss of performance for a much desired capability
offered by an addon, in such cases the said system can contribute to
an informed decision-making process.
If the tools are easier to use, some of us can test it and post the
results to an AMO back-end, which can then be made available to the
users through AMO. There can even be a trusted status or voluntary
moderator-only access to prevent miscued results.