A summary of ideas to improve add-ons

[Nicholas Nethercote]

Hi,

There's been a lot of discussion about add-ons lately, and lots of
ideas suggested on how to improve them. I've attempted to summarize
the problems and the ideas in this etherpad:

https://etherpad.mozilla.org/M3nBRlRiXF

The three problem areas are "Add-ons can hurt performance", "Non-AMO
add-ons have no minimal quality standard", "Foreign-installed add-ons
can be unwanted". There is some overlap between the first two areas
(performance is an aspect of quality) but I felt they were worth
separating.

Some of the listed ideas are already in place. Of those that are not,
I think the following are both worth doing and likely to be
uncontroversial:

- offer reviews/QA for non-AMO add-ons
- provide listings for non-AMO add-ons on AMO

(Just because an idea is controversial doesn't mean it's bad. But
given how much disagreement there has been about this stuff lately, I
thought it worth highlighting these ideas right from the start :)

Nick

[Henri Sivonen]

I've noticed that @firefox on Twitter has promoted at least one add-on
that you list as having leaks (Lastpass). Maybe we should have some
rule of not promoting foreign-installed add-ons or add-ons with known
problems or add-ons that replace features (like password manager or
safe browsing) that Firefox already has built-in.

--
Henri Sivonen
hsiv...@iki.fi
http://hsivonen.iki.fi/

[Michael Lefevre]

On 15/03/2012 10:23, Henri Sivonen wrote:
> I've noticed that @firefox on Twitter has promoted at least one add-on
> that you list as having leaks (Lastpass). Maybe we should have some
> rule of not promoting foreign-installed add-ons or add-ons with known
> problems or add-ons that replace features (like password manager or
> safe browsing) that Firefox already has built-in.

I know you've used "or", but that starts to look like a way of never
promoting any add-ons at all.

Lastpass might be foreign-installed as well, but it is also on AMO,
fully reviewed and listed as a "featured" addon. Actually I can
immediately see several add-ons on the AMO featured list which are also
on the "leaking" list, and which replace built-in functions.

If users get told too much that add-ons are bad and hurt performance,
then they're not going to want any add-ons.

Additionally, Lastpass (and similar) have been promoted (I think within
Mozilla forums as well as elsewhere) specifically because the built-in
feature is lacking in some areas and are no longer being worked on
(which gives me an opportunity to complain about bug 355063 - Password
manager does not work on script-generated forms - which affects more and
more sites).

Michael

[Henri Sivonen]

On Thu, Mar 15, 2012 at 1:05 PM, Michael Lefevre
<mjl+...@michaellefevre.com> wrote:
> On 15/03/2012 10:23, Henri Sivonen wrote:
>>
>> I've noticed that @firefox on Twitter has promoted at least one add-on
>> that you list as having leaks (Lastpass). Maybe we should have some
>> rule of not promoting foreign-installed add-ons or add-ons with known
>> problems or add-ons that replace features (like password manager or
>> safe browsing) that Firefox already has built-in.
>
> I know you've used "or", but that starts to look like a way of never
> promoting any add-ons at all.

Surely there are non-leaky add-ons on AMO that don't replace built-in
functionality.

> Lastpass might be foreign-installed as well, but it is also on AMO, fully
> reviewed and listed as a "featured" addon. Actually I can immediately see
> several add-ons on the AMO featured list which are also on the "leaking"
> list, and which replace built-in functions.

I think leaky add-ons shouldn't be on the features list, either.

[Justin Lebar]

>> Lastpass might be foreign-installed as well, but it is also on AMO, fully
>> reviewed and listed as a "featured" addon. Actually I can immediately see
>> several add-ons on the AMO featured list which are also on the "leaking"
>> list, and which replace built-in functions.
>

> I think leaky add-ons shouldn't be on the features list, either.

Agreed.

But leaky add-ons shouldn't be on AMO at all.

This is something we've basically agreed on, and we're moving in that
direction. All new add-ons are reviewed for leaks; add-ons which leak
are not approved for AMO. When we find a leak in an add-on
post-review (or in an add-on which was approved before we started
testing for leaks), we contact the author and, if the add-on is not
fixed in a timely fashion, we downgrade the add-on to preliminary
status on AMO.

This has been the topic of some heroic work by the add-ons folks. A
lot of issues have been fixed, and we're just getting started.


Instead of simply downgrading to preliminary, I think we should

a) Say on the AMO page that the add-on is known to leak -- the
preliminary review warning is not particularly informative [1] -- and
b) Notify all users who have the add-on installed that it may be
causing problems.

There was originally some question as to whether (a) and (b) would be
necessary. It might have been that we'd have to downgrade very few
add-ons, because most authors would be responsive and fix their bugs.
I think we've observed now that this is not the case. Despite plenty
of bugs fixed, we've also downgraded some popular add-ons [2]. So I
hope we'll get to work on (a) and (b) above.

[1] An add-on in preliminary review looks like:
https://addons.mozilla.org/en-US/firefox/addon/baow/

[2] Downgraded add-ons that I'm aware of include:
https://bugzilla.mozilla.org/show_bug.cgi?id=728528 <-- 250 users.
Not a popular add-on. :)
https://bugzilla.mozilla.org/show_bug.cgi?id=691102 <-- ~50th most
popular AMO add-on, 400,000 users, compatible through FF 11
https://bugzilla.mozilla.org/show_bug.cgi?id=716163 <-- 90th most
popular AMO add-on, 150,000 users, compatible through FF 11

[Kyle Huey]

On Thu, Mar 15, 2012 at 3:23 AM, Henri Sivonen <hsiv...@iki.fi> wrote:

> I've noticed that @firefox on Twitter has promoted at least one add-on
> that you list as having leaks (Lastpass). Maybe we should have some
> rule of not promoting foreign-installed add-ons or add-ons with known
> problems or add-ons that replace features (like password manager or
> safe browsing) that Firefox already has built-in.

What's wrong with addons replacing built-in features, if the replacement is
at the same level of quality (in terms of leaks/perf/etc)?

- Kyle

[Henri Sivonen]

It sets users up to grief like what was seen when Google Toolbar went
away and people had their bookmarks there. Having passwords in a
3rd-party password manager is a setup for a situation where user
workflow will be badly disrupted if the extension stops working.
Outright *promoting* such extensions sets us up for having problems
with "too disruptive to fail" add-ons.

As for safe browsing replacements, are they really same quality in
terms of leaks and perf as the built-in feature?

[Jorge Villalobos]

On 3/15/12 4:23 AM, Henri Sivonen wrote:
> I've noticed that @firefox on Twitter has promoted at least one add-on
> that you list as having leaks (Lastpass).

The memory leak hunt is only getting started, so we're still in a
discovery phase where many add-on leaks will be discovered. We're only
beginning to inform developers how to look for leaks and fix them. While
some are the result of bad coding practices, others can be caused by
minor oversights (like an undeclared variable).

I expect many popular add-ons to have leaks, since they usually have the
most complex code, and given that developers haven't been actively
looking for them since they don't know how to.

> Maybe we should have some
> rule of not promoting foreign-installed add-ons

We only promote AMO add-ons as far as I know.

> or add-ons with known problems

Depending on the severity of the problem, we already have this policy.
We don't promote add-ons that are not fully reviewed, and we work with
the Featured Add-ons group to make sure new add-ons being featured
aren't problematic.

> or add-ons that replace features (like password manager or
> safe browsing) that Firefox already has built-in.

I disagree with this completely.

- Jorge

[Jorge Villalobos]

On 3/15/12 4:23 AM, Henri Sivonen wrote:

> I've noticed that @firefox on Twitter has promoted at least one add-on
> that you list as having leaks (Lastpass).

The memory leak hunt is only getting started, so we're still in a
discovery phase where many add-on leaks will be discovered. We're only
beginning to inform developers how to look for leaks and fix them. While
some are the result of bad coding practices, others can be caused by
minor oversights (like an undeclared variable).

I expect many popular add-ons to have leaks, since they usually have the
most complex code, and given that developers haven't been actively
looking for them since they don't know how to.

> Maybe we should have some
> rule of not promoting foreign-installed add-ons

We only promote AMO add-ons as far as I know.

> or add-ons with known problems

Depending on the severity of the problem, we already have this policy.
We don't promote add-ons that are not fully reviewed, and we work with
the Featured Add-ons group to make sure new add-ons being featured
aren't problematic.

> or add-ons that replace features (like password manager or
> safe browsing) that Firefox already has built-in.

[Dave Townsend]

On 03/14/12 16:36, Nicholas Nethercote wrote:
> - provide listings for non-AMO add-ons on AMO

We used to do this, I don't recall why we stopped.

[Michael Lefevre]

On 15/03/2012 14:55, Henri Sivonen wrote:
> On Thu, Mar 15, 2012 at 4:52 PM, Kyle Huey<m...@kylehuey.com> wrote:

...

>> What's wrong with addons replacing built-in features, if the replacement is
>> at the same level of quality (in terms of leaks/perf/etc)?
>
> It sets users up to grief like what was seen when Google Toolbar went
> away and people had their bookmarks there. Having passwords in a
> 3rd-party password manager is a setup for a situation where user
> workflow will be badly disrupted if the extension stops working.
> Outright *promoting* such extensions sets us up for having problems
> with "too disruptive to fail" add-ons.

My understanding of this was that there was a benefit to Mozilla in not
having to develop and maintain the additional fixes/features that these
add-ons provide. If there's a problem, then rather than trying to hide
their existence from people, an obvious (and more positive) solution
would be to make the fixes and add the features that (some) people want
to the built-in facility.

Adding a bunch of features and fixes to the download manager, password
manager and tabbed browsing could wipe out the popularity of several
potentially "too disruptive to fail" add-ons pretty quickly.

Michael

[Nicholas Nethercote]

On Thu, Mar 15, 2012 at 9:28 AM, Jorge Villalobos <jo...@mozilla.com> wrote:
>
>> Maybe we should have some
>> rule of not promoting foreign-installed add-ons
>
> We only promote AMO add-ons as far as I know.
>
>> or add-ons with known problems
>
> Depending on the severity of the problem, we already have this policy. We
> don't promote add-ons that are not fully reviewed, and we work with the
> Featured Add-ons group to make sure new add-ons being featured aren't
> problematic.
>
>> or add-ons that replace features (like password manager or
>> safe browsing) that Firefox already has built-in.
>
> I disagree with this completely.

This all sounds fine to me.

Can we not get too distracted by the issue of promoted add-ons? I
think they are a very minor part of this overall discussion.

Nick

[Nicholas Nethercote]

On Thu, Mar 15, 2012 at 7:39 AM, Justin Lebar <justin...@gmail.com> wrote:
>
> a) Say on the AMO page that the add-on is known to leak -- the
> preliminary review warning is not particularly informative [1]

How are preliminarily reviewed add-ons distinguished on AMO? I can
see they have an easily-missed bit of text under the big "Add to
Firefox" button that says:

This add-on has been preliminarily reviewed by Mozilla. Learn More.

The "Learn More" link says this:

What does it mean if an add-on is "experimental" or "preliminarily reviewed"?
Experimental add-ons have been checked by our editors to make sure
they don't have security problems, but they may still have bugs or not
work properly. Use caution when installing experimental add-ons and
uninstall the add-on immediately if you notice problems.

which doesn't actually explain what "preliminarily reviewed" means.
There's a "Learn more about our review process" link. If you follow
that you get to
https://addons.mozilla.org/en-US/developers/docs/policies/reviews,
which finally explains what it means ("preliminarily reviewed" and
"experimental" are basically synonyms, AFAICT).

Are preliminarily reviewed add-ons distinguished in any other way on AMO?

Nick

[Jorge Villalobos]

http://blog.mozilla.com/addons/2010/10/06/discontinuing-several-features-of-amo/

[Jorge Villalobos]

* You'll see a second warning when you click on the Add to Firefox button.

* On search results, fully reviewed add-ons will always rank higher.

* They are never featured, so they won't show up in the most prominent
places (except maybe in the most popular rankings, if they are very
popular).

* The button has a different color (yellow with stripes instead of green).

- Jorge

[Jorge Villalobos]

On 3/15/12 4:52 PM, Nicholas Nethercote wrote:

[Jorge Villalobos]

On 3/15/12 5:07 PM, Jorge Villalobos wrote:
> * You'll see a second warning when you click on the Add to Firefox button.

Scratch this. That is only true for unreviewed add-ons, not
preliminarily reviewed ones.

- Jorge

[Jorge Villalobos]

On 3/15/12 5:07 PM, Jorge Villalobos wrote:

> * You'll see a second warning when you click on the Add to Firefox button.

[Nicholas Nethercote]

On Thu, Mar 15, 2012 at 4:02 PM, Jorge Villalobos <jo...@mozilla.com> wrote:
>>>
>>> - provide listings for non-AMO add-ons on AMO
>>
>> We used to do this, I don't recall why we stopped.
>
> http://blog.mozilla.com/addons/2010/10/06/discontinuing-several-features-of-amo/

Oh, that's interesting, thanks for the link. The key quote seems to be this:

"many of the best add-ons that aren’t hosted on AMO did not
participate. Instead, add-ons that were already listed on AMO switched
to self-hosted to avoid complying with certain review policies."

Unintended side-effects! I guess the visibility benefits of being on
AMO are high enough that it makes it worthwhile to go through the
hurdle of the review process.

Nick

[Asa Dotzler]

I think making it as you first said it was would be a good step forward.
Can we do that? I'll file the bug.

- A

[Asa Dotzler]

On 3/14/2012 4:36 PM, Nicholas Nethercote wrote:
> The three problem areas are

> "Add-ons can hurt performance"

This is the largest problem we face today with add-ons. I count in
"performance" everything from start-up time to GUI and page-load
performance to jank caused by excessive memory usage. Any real solution
to this problem must either prevent add-ons with these kinds of problems
from ever reaching our users or must ensure that our users have real,
actionable information about the problems the add-on can cause before
installing (for new installations) and after installation (for
installations that were in place before the new warning.)

> "Non-AMO add-ons have no minimal quality standard"

Serious problems in this area which aren't covered by performance are
stability, Firefox UX breakage, and site breakage. If the add-on's
features are not well designed for usability, that is a problem, but if
the add-on doesn't cause problems with Firefox and is only a poor
experience in and of itself, I think that's less serious. As with the
first category, any thorough solution to this problem should prevent
add-ons which cause Firefox crashes, or UX or site breakage from
reaching users or must ensure that our users have real, actionable
information about the problems.

> "Foreign-installed add-ons can be unwanted"

If the problems of performance and quality are addressed, either by
preventing those problems from reaching users or by informing users
about the problems so they can make good choices, this area should be
mostly limited to privacy, security, and usability problems. I believe
that we have solved some of this problem with prompts to warn users of
foreign-installed add-ons. I'm concerned that the implementation there
is not as usable/actionable as it could be and I'd like us to invest
more in UX/UR to get that as good as it can be. We also can mitigate
with easier tools for toggling on and off foreign-installed toolbars.
The Australis Firefox re-design includes plans to make this easier.

> Some of the listed ideas are already in place. Of those that are
not, I think the following are both worth doing and likely to be
uncontroversial:

> - offer reviews/QA for non-AMO add-ons

I think we should make this available and we should do pro-active
outreach to push this onto non-AMO add-on authors -- at least any with
substantial user numbers.

> - provide listings for non-AMO add-ons on AMO

This seems like an absolute necessity for providing users with real
choice. Listings, review states, and ratings are all necessary. It's the
only way we can properly warn them about shortcomings with add-ons. We
should surface this information during foreign-install "are you sure"
prompts and we should make sure it's easily accessible for add-ons
installed from non-AMO websites. For users who already have add-ons
installed, we should offer a one-time prompt on a future update that
helps them understand the implications of their current add-ons. We
should also update the add-on manager to grab relevant data from the
listings and display it along side the add-on in the add-ons manager.

- A

[Mark Finkle]

On 03/15/2012 06:23 AM, Henri Sivonen wrote:

> add-ons that replace features (like password manager or
> safe browsing) that Firefox already has built-in.

I also disagree with this idea. It's one of the reasons add-ons are
popular in the first place.

[Gijs Kruitbosch]

On 18/03/2012 01:58 AM, Asa Dotzler wrote:
> <snip>

> > - provide listings for non-AMO add-ons on AMO
>
> This seems like an absolute necessity for providing users with real choice.
> Listings, review states, and ratings are all necessary. It's the only way we can
> properly warn them about shortcomings with add-ons. We should surface this
> information during foreign-install "are you sure" prompts and we should make
> sure it's easily accessible for add-ons installed from non-AMO websites. For
> users who already have add-ons installed, we should offer a one-time prompt on a
> future update that helps them understand the implications of their current
> add-ons. We should also update the add-on manager to grab relevant data from the
> listings and display it along side the add-on in the add-ons manager.
>
> - A

How do we do this? I mean, the add-ons might or might not be signed. To the best
of my knowledge, add-on IDs are not secret and anyone can theoretically use any
other ID. Couldn't they just use anyone's add-on ID (say, one that's
well-rated/reviewed)? Or keep switching IDs?

I'm not even entirely sure how we'd go about 'authenticating' an add-on -- even
if we force signing them, we'd need to start correlating the signatures with the
IDs in some way... (which, to the best of my knowledge, they currently aren't).

~ Gijs

[Asa Dotzler]

Like that Google Toolbar that so helpfully replaced your Firefox
bookmarks and history and passwords -- until they discontinued it and
you lost everything.

Users should understand what they're getting into with add-ons that
replace Firefox functionality like that. I don't think most of them do.

- A

[Nicholas Nethercote]

On Sun, Mar 18, 2012 at 11:58 AM, Asa Dotzler <a...@mozilla.org> wrote:
>>
>> - provide listings for non-AMO add-ons on AMO
>
> This seems like an absolute necessity for providing users with real choice.
> Listings, review states, and ratings are all necessary. It's the only way we
> can properly warn them about shortcomings with add-ons. We should surface
> this information during foreign-install "are you sure" prompts and we should
> make sure it's easily accessible for add-ons installed from non-AMO
> websites. For users who already have add-ons installed, we should offer a
> one-time prompt on a future update that helps them understand the
> implications of their current add-ons. We should also update the add-on
> manager to grab relevant data from the listings and display it along side
> the add-on in the add-ons manager.

Jorge cited http://blog.mozilla.com/addons/2010/10/06/discontinuing-several-features-of-amo/,
which explains how AMO used to do this. The problem is that it
perverts the incentives -- add-on authors can get the benefits of
being on AMO (visibility) without the costs (having to pass review).
This makes it less likely that authors will distribute their add-ons
through AMO, which is bad.

Maybe the model could be tweaked to avoid this problem -- e.g. allow
reviews and a description of an add-on without providing a download
link? But that sounds a bit odd. The link also explains that it made
AMO significantly more complex.

Nick

[Asa Dotzler]

I mostly care about this data being available so that when a user
installs an add-on, we can slurp it into our installation dialog and
after we can include it in the add-on manager. We don't have to expose
all (or any) of it at AMO.

- A

[Mark Finkle]

I would argue that point for all add-ons.

[Dave Townsend]

We have most of the infrastructure in place for this already and we did
the one-time prompt with Firefox 8. What new information would you
expect us to add to the listings for add-ons on AMO that you think would
help users make their choice?

[Asa Dotzler]

Performance and quality information for sure. Perhaps warnings about the
problem of putting Firefox core features into the hands of third parties
which may not be around for very long.

- A

[Asa Dotzler]

I don't think it's worth trying to explain to users, for most add-ons,
that if they lose the add-on they'll lose that add-on's functionality.
For most add-ons, that's kind of obvious. For add-ons that replace
Firefox functionality, though, that loss is entirely different and far
more consequential.

- A

[Blair McBride]

(Apologies for being late to the party - I got swamped, then I got sick;
I'm now catching up between naps.)

On 18/03/2012 9:20 p.m., Nicholas Nethercote wrote:
> Jorge cited http://blog.mozilla.com/addons/2010/10/06/discontinuing-several-features-of-amo/,
> which explains how AMO used to do this. The problem is that it
> perverts the incentives -- add-on authors can get the benefits of
> being on AMO (visibility) without the costs (having to pass review).
> This makes it less likely that authors will distribute their add-ons
> through AMO, which is bad.
>
> Maybe the model could be tweaked to avoid this problem -- e.g. allow
> reviews and a description of an add-on without providing a download
> link? But that sounds a bit odd. The link also explains that it made
> AMO significantly more complex.

That model doesn't fit with the problem at hand anyway, since it
primarily solves discoverability.

Add-ons don't need to be *listed* on AMO for us to get the benefits
(reviews, rating, etc). Nor do they need to be submitted. The problem
here isn't discoverability/visibility of non-hosted add-ons.

All we need is storage of reviews/ratings/etc, and an API. And
consumption of that data is contextual:
* When you're about to install such an addon
* When you already have such an add-on installed

And that's *any* non-hosted addon, not just those that were added to
AMO's database by the addon author.

When we install/detect an addon, we already ask AMO for additional
metadata - we can make that include reviews/ratings/etc for non-hosted
addons. That ping also gives us a list of non-hosted add-ons to
review/test/etc (which has already started).

- Blair

[Nicholas Nethercote]

On Tue, Mar 20, 2012 at 9:43 PM, Blair McBride <bmcb...@mozilla.com> wrote:
>
> That model doesn't fit with the problem at hand anyway, since it primarily
> solves discoverability.
>
> Add-ons don't need to be *listed* on AMO for us to get the benefits
> (reviews, rating, etc). Nor do they need to be submitted. The problem here
> isn't discoverability/visibility of non-hosted add-ons.
>
> All we need is storage of reviews/ratings/etc, and an API. And consumption
> of that data is contextual:
> * When you're about to install such an addon
> * When you already have such an add-on installed
>
> And that's *any* non-hosted addon, not just those that were added to AMO's
> database by the addon author.
>
> When we install/detect an addon, we already ask AMO for additional metadata
> - we can make that include reviews/ratings/etc for non-hosted addons. That
> ping also gives us a list of non-hosted add-ons to review/test/etc (which
> has already started).

That's all true, but if an add-on isn't listed on AMO how/where do
users submit reviews and ratings?

Nick

[Jorge Villalobos]

In the Add-ons Manager?

* ducks *

- Jorge

[Jorge Villalobos]

On 3/20/12 5:30 AM, Nicholas Nethercote wrote:

[Dave Townsend]

The UI can go there sure, we need a service on the web somewhere to
aggregate that data though. AMO already has this functionality.

[Nicholas Nethercote]

On Tue, Mar 20, 2012 at 12:49 PM, Dave Townsend <dtow...@mozilla.com> wrote:
>>>
>>> That's all true, but if an add-on isn't listed on AMO how/where do
>>> users submit reviews and ratings?
>>

>> In the Add-ons Manager?
>
> The UI can go there sure, we need a service on the web somewhere to
> aggregate that data though. AMO already has this functionality.

Regardless of whether the info is presented on AMO or in the add-ons
manager, I imagine users will want to be able to see their reviews and
ratings somewhere public and obvious -- it's not much fun to submit
reviews/ratings somewhere if those reviews/ratings are only shown to
people who install an add-on. (But I could be wrong about this...)
But as soon as we make the reviews/ratings public and obvious we're
back to the "free visibility for unreviewed add-ons" problem.

Nick

[Nicholas Nethercote]

On Tue, Mar 20, 2012 at 5:53 PM, Nicholas Nethercote
<n.neth...@gmail.com> wrote:
>
> Regardless of whether the info is presented on AMO or in the add-ons
> manager, I imagine users will want to be able to see their reviews and
> ratings somewhere public and obvious -- it's not much fun to submit
> reviews/ratings somewhere if those reviews/ratings are only shown to
> people who install an add-on.  (But I could be wrong about this...)

Hmm, but maybe if we had "this add-on is great" / "this add-on sucks"
buttons in the add-on manager (or maybe a "rate this add-on from 1 to
5 stars" button) that would make rating easy enough that the user
wouldn't expect anything once they hit "submit". We could allow a
short description to go along with the rating as well. It'd be a bit
like the Firefox feedback add-on thingy that is (was?) in beta
versions.

Nick

[Blair McBride]

Yep. I think it only needs to be something lightweight. I don't think
it'd be terribly be useful to let someone see that at 7.13pm last
Tuesday they rated Addon Y as "2 stars". It's more useful to show that
Addon Y isn't very good and you probably shouldn't install it.

- Blair