Proposal for whitelisting add-ons for review

[Jorge Villalobos]

Hello,

This came up in a couple of previous threads that are pretty deep
already, so I'm starting a new thread so we can focus on this topic. The
proposal is that we create a mechanism where AMO auto-approves and signs
add-ons under certain circumstances. We already do this for unlisted
add-ons that don't raise serious validation flags, but the proposal is
to extend the criteria to add-ons that would never be able to pass the
minimum validation bar.

So far we have one set of conditions proposed by Gijs.

> How about add-ons with all of:
> - more than 100,000 active daily users
> - at least 1 year of history without serious review issues as indicated by the AMO reviewer team
> - rapid releases (figure out some way to measure this, vis-a-vis review latency)
> - source code that's publicly available elsewhere than just via the AMO viewer itself, and a public issue tracker (bugzilla, github issues, whatever)
> - no AMO policy violations
>
> can apply to be on it, with the caveat that the whitelist isn't a "right" and the AMO team can remove add-ons that start violating any of these, even if they weren't when they were "admitted" to the whitelist.

I think they are a very good starting point. Some comments of my own:

> - more than 100,000 active daily users

This number seems reasonable, and we do have approximate usage data for
non-AMO add-ons (this question was brought up before).

> - at least 1 year of history without serious review issues as indicated by the AMO reviewer team

I think it can be less than a year. I could be something like "the last
6 months if there are 5 releases or more during that time, otherwise the
last 5 releases".

However, this introduces a vulnerability, which is that add-ons can
easily be bought out or otherwise change hands, without us knowing.
We've seen this happen on AMO many times, and add-ons that used to be
excellent introduce unwanted and sometimes abusive features, and quickly
burn out. We could introduce a requirement that ownership changes reset
the whitelisting status, but that would depend on the good faith of the
original developers (or even if they remember this agreement).

Determining what constitutes a "serious review issue" is also important.
Having a willingness to make corrections that make an add-on more
reviewable by us is very important, and whitelisting isn't a free pass
to make an add-on even more inscrutable. This is the main conflict in
this proposal, I think. The developers who would most like to be
whitelisted are the same ones that would be regularly rejected because
the code has a myriad of issues, is completely obfuscated and don't want
to share any sources, or have massive codebases that take lots of time
to look over.

> - rapid releases (figure out some way to measure this, vis-a-vis review latency)

I don't think this is a great metric, since it's trivial to game and
some add-ons that would request whitelisting might not be updated that
often, but would still have a strong need to not be blocked by the
review process.

> - source code that's publicly available elsewhere than just via the AMO viewer itself, and a public issue tracker (bugzilla, github issues, whatever)

That sounds good to me.

> - no AMO policy violations

This would be considered a serious review issue, so I think it's redundant.

> the whitelist isn't a "right" and the AMO team can remove add-ons that start violating any of these

And we can remove add-ons at our own discretion, if we decide that
whitelisting isn't a good idea after all, or if we believe the review
process is good enough that we don't need whitelisting anymore.

I would also add that add-on developers should be ready to have their
add-on reviewed at any point in time. While their versions may be
auto-approved, the review team will need to give them a look every now
and then.

I also think we should make the whitelist flag temporary, so it is
re-evaluated on a yearly basis.

Jorge

[Dan Stillman]

On 11/17/15 5:55 PM, Jorge Villalobos wrote:
> This came up in a couple of previous threads that are pretty deep
> already, so I'm starting a new thread so we can focus on this topic.

Thanks, Jorge.

>> - more than 100,000 active daily users
> This number seems reasonable, and we do have approximate usage data for
> non-AMO add-ons (this question was brought up before).

What's the rationale for using ADU as a metric? As I noted earlier, an
extension with relatively fewer users might still have an appropriately
conscientious and dedicated developer and still be critical to users'
workflows (e.g., Emiliano's extension), and having a small user base
means that the risk of a (very) hypothetical problem is much smaller anyway.

(Also, where does the approximate usage data for non-AMO add-ons come from?)

>> - at least 1 year of history without serious review issues as indicated by the AMO reviewer team
> I think it can be less than a year. I could be something like "the last
> 6 months if there are 5 releases or more during that time, otherwise the
> last 5 releases".
>
> However, this introduces a vulnerability, which is that add-ons can
> easily be bought out or otherwise change hands, without us knowing.
> We've seen this happen on AMO many times, and add-ons that used to be
> excellent introduce unwanted and sometimes abusive features, and quickly
> burn out. We could introduce a requirement that ownership changes reset
> the whitelisting status, but that would depend on the good faith of the
> original developers (or even if they remember this agreement).

Yes, I was going to suggest that the original developer could be
required to notify Mozilla upon ownership changes, which, if honored
(admittedly a big if), would at least give Mozilla the opportunity to
reconsider whitelist status. No idea if this is possible, but maybe
there's some way under contract law to hold the original developer
liable for damages (time writing cleanup code?) if they fail to provide
notification and the transferred extension is abused.

But also, how is seeing this on AMO many times relevant to unlisted
add-ons? It seems like you're admitting that even AMO review hasn't
prevented this sort of behavior, which rather undercuts the notion that
unreviewed extensions are a much greater danger. In any case, the more
relevant metric would be how many unlisted, front-loaded extensions
you've seen this happen to — extensions where the developer is generally
much more personally vouching for the extension than on AMO. Named
examples of those would be helpful so that we can evaluate whether the
proposed rules would have weeded them out.

- Dan

[David E. Ross]

On 11/17/2015 2:55 PM, Jorge Villalobos wrote [in part]:

[snipped]

>> - more than 100,000 active daily users
>
> This number seems reasonable, and we do have approximate usage data for
> non-AMO add-ons (this question was brought up before).

[snipped]

How is this measured? Is Mozilla snooping into my Web surfing?

--
David E. Ross

The Crimea is Putin's Sudetenland.
The Ukraine will be Putin's Czechoslovakia.
See <http://www.rossde.com/editorials/edtl_PutinUkraine.html>.

[Jorge Villalobos]

On 11/18/15 9:57 AM, David E. Ross wrote:
> On 11/17/2015 2:55 PM, Jorge Villalobos wrote [in part]:
>
> [snipped]
>
>>> - more than 100,000 active daily users
>>
>> This number seems reasonable, and we do have approximate usage data for
>> non-AMO add-ons (this question was brought up before).
>
> [snipped]
>
> How is this measured? Is Mozilla snooping into my Web surfing?
>

It's obtained via the Firefox Health Report:
https://www.mozilla.org/en-US/privacy/firefox/#health-report.

It doesn't report browsing data, and you can see what's being shared in
about:healthreport

Jorge

[Jorge Villalobos]

On 11/18/15 1:36 AM, Dan Stillman wrote:
> On 11/17/15 5:55 PM, Jorge Villalobos wrote:
>> This came up in a couple of previous threads that are pretty deep
>> already, so I'm starting a new thread so we can focus on this topic.
>
> Thanks, Jorge.
>
>>> - more than 100,000 active daily users
>> This number seems reasonable, and we do have approximate usage data for
>> non-AMO add-ons (this question was brought up before).
>
> What's the rationale for using ADU as a metric? As I noted earlier, an
> extension with relatively fewer users might still have an appropriately
> conscientious and dedicated developer and still be critical to users'
> workflows (e.g., Emiliano's extension), and having a small user base
> means that the risk of a (very) hypothetical problem is much smaller
> anyway.

While the risk may be smaller, the potential gain for our users is also
smaller. I think that keeping a small whitelist is important, and the
usage metric is a useful and fairly objective filter for this.

> (Also, where does the approximate usage data for non-AMO add-ons come
> from?)

FHR, see my response to David E. Ross in this thread.

>
>>> - at least 1 year of history without serious review issues as
>>> indicated by the AMO reviewer team
>> I think it can be less than a year. I could be something like "the last
>> 6 months if there are 5 releases or more during that time, otherwise the
>> last 5 releases".
>>
>> However, this introduces a vulnerability, which is that add-ons can
>> easily be bought out or otherwise change hands, without us knowing.
>> We've seen this happen on AMO many times, and add-ons that used to be
>> excellent introduce unwanted and sometimes abusive features, and quickly
>> burn out. We could introduce a requirement that ownership changes reset
>> the whitelisting status, but that would depend on the good faith of the
>> original developers (or even if they remember this agreement).
>
> Yes, I was going to suggest that the original developer could be
> required to notify Mozilla upon ownership changes, which, if honored
> (admittedly a big if), would at least give Mozilla the opportunity to
> reconsider whitelist status. No idea if this is possible, but maybe
> there's some way under contract law to hold the original developer
> liable for damages (time writing cleanup code?) if they fail to provide
> notification and the transferred extension is abused.

I doubt that's possible, and if it were, it would probably require
signing some kind of contract, which would make this system more
difficult to implement and maintain.

> But also, how is seeing this on AMO many times relevant to unlisted
> add-ons?

It's relevant as evidence that this sort of thing happens regularly.

> It seems like you're admitting that even AMO review hasn't
> prevented this sort of behavior, which rather undercuts the notion that
> unreviewed extensions are a much greater danger.

Some of these cases weren't policy violations, but were a significant
turn in the way the add-on and developers behaved, causing grief to
users. There are admittedly cases where the developer broke a policy and
a reviewer didn't notice the change, possibly because of the impeccable
review history the add-on previously had. We implemented a way to flag
ownership changes since, but that only works when the developer account
associated to the add-on changes, not when the original developer
account is given to a new person or group.

Unlisted add-ons are riskier because they have to pass a lower admission
bar than listed ones, so the changes that can be added in an update
after an ownership change can cause more problems.

> In any case, the more
> relevant metric would be how many unlisted, front-loaded extensions
> you've seen this happen to — extensions where the developer is generally
> much more personally vouching for the extension than on AMO. Named
> examples of those would be helpful so that we can evaluate whether the
> proposed rules would have weeded them out.
>
> - Dan

We have very little historical information about unlisted add-ons. We
have blocked numerous unlisted, probably front-loaded extensions along
the years, but they are mostly of malicious nature. I don't recall a
case of ownership changes being the cause, but it's unlikely we would
even hear about that. There might be cases in the blocklisted add-ons
history (https://addons.mozilla.org/firefox/blocked/), but it's pretty
long and I don't have the time to dig and search for a concrete example.

Jorge

[David E. Ross]

Oh! I was told that Firefox Health Report was not implemented in
SeaMonkey, my preferred browser. Thus, the proposed whitelist criteria
would discriminate against SeaMonkey users.

[Dan Stillman]

On 11/18/15 12:56 PM, Jorge Villalobos wrote:
> On 11/18/15 1:36 AM, Dan Stillman wrote:

>> What's the rationale for using ADU as a metric? As I noted earlier, an
>> extension with relatively fewer users might still have an appropriately
>> conscientious and dedicated developer and still be critical to users'
>> workflows (e.g., Emiliano's extension), and having a small user base
>> means that the risk of a (very) hypothetical problem is much smaller
>> anyway.

> While the risk may be smaller, the potential gain for our users is also
> smaller. I think that keeping a small whitelist is important, and the
> usage metric is a useful and fairly objective filter for this.

The risk and gain are both limited to precisely the number of users of
the extension. It seems like the fair and objective metric here to keep
a small whitelist would be "difficulty in passing automated review",
perhaps with a slightly less objective additional metric of "importance
of timely updates". If someone can already pass automated review, adding
them to a whitelist doesn't seem necessary — if rapid updates are
important, they can add the signing API or offline validator to their CI
pipeline so that they're notified of new failures.

This metric isn't our fight, and I'm actually not sure if Emiliano is
even having trouble passing automated review at the moment, but given
that Zotero and Emiliano are so far the only developers to even come
forward and articulate the need for a whitelist, it's not clear that
more than a small number of developers are in this position.

>> But also, how is seeing this on AMO many times relevant to unlisted
>> add-ons?

> It's relevant as evidence that this sort of thing happens regularly.

Well, it's not really evidence if you don't provide concrete examples
that we can evaluate against the proposed rules, but at most it's
evidence that these sorts of sales happen regularly for AMO extensions.
I'm arguing that there's much less reason to think they happen regularly
for unlisted extensions, where users need to come to a site and
personally trust a developer.

In any case, I hope we can agree that it's even less likely to happen in
the case of whitelisted extensions, which would pass an even higher bar
of trust and in most cases (by virtue of the public bug tracker
requirement, among other things) have a community around them.

> We have very little historical information about unlisted add-ons. We
> have blocked numerous unlisted, probably front-loaded extensions along
> the years, but they are mostly of malicious nature. I don't recall a
> case of ownership changes being the cause, but it's unlikely we would
> even hear about that. There might be cases in the blocklisted add-ons
> history (https://addons.mozilla.org/firefox/blocked/), but it's pretty
> long and I don't have the time to dig and search for a concrete example.

OK, but just to be clear, then still no one can provide an example of a
front-loaded, unlisted extension that was legitimate and then became
illegitimate. I'm not denying that there might have been some, or that
it's possible, but I don't think we can draw any conclusion other than
that it's not a problem that merits significant concern — and that
whitelisted extensions would be even less likely to be a problem. We
need to weigh the highly unlikely specter of that against the very real
needs that the whitelist is trying to address.

In any case, other than the ADU metric, the current proposal seems
reasonable to me.

- Dan

[Jorge Villalobos]

On 11/18/15 12:11 PM, David E. Ross wrote:
> On 11/18/2015 8:22 AM, Jorge Villalobos wrote:
>> On 11/18/15 9:57 AM, David E. Ross wrote:
>>> On 11/17/2015 2:55 PM, Jorge Villalobos wrote [in part]:
>>>
>>> [snipped]
>>>
>>>>> - more than 100,000 active daily users
>>>>
>>>> This number seems reasonable, and we do have approximate usage data for
>>>> non-AMO add-ons (this question was brought up before).
>>>
>>> [snipped]
>>>
>>> How is this measured? Is Mozilla snooping into my Web surfing?
>>>
>>
>> It's obtained via the Firefox Health Report:
>> https://www.mozilla.org/en-US/privacy/firefox/#health-report.
>>
>> It doesn't report browsing data, and you can see what's being shared in
>> about:healthreport
>>
>> Jorge
>>
>
> Oh! I was told that Firefox Health Report was not implemented in
> SeaMonkey, my preferred browser. Thus, the proposed whitelist criteria
> would discriminate against SeaMonkey users.
>

Add-ons that are exclusive to SeaMonkey don't need to be signed. For
most Firefox add-ons, SeaMonkey usage would be a very small fraction of
total usage (for example:
https://addons.mozilla.org/addon/adblock-plus/statistics/usage/applications/?last=30),
so I don't think this is a big concern. I guess if a dev had ~99K users
and argued the other 1K are SM users, we could take their word for it.
Not a big deal.

Jorge

[Mike Connor]

(Today's a bit nutty, I'll catch up on this thread with more extensive
comments as soon as I can.)

On Wed, Nov 18, 2015 at 2:42 PM, Dan Stillman <dsti...@zotero.org> wrote:

>
> OK, but just to be clear, then still no one can provide an example of a
> front-loaded, unlisted extension that was legitimate and then became
> illegitimate. I'm not denying that there might have been some, or that it's
> possible, but I don't think we can draw any conclusion other than that it's
> not a problem that merits significant concern — and that whitelisted
> extensions would be even less likely to be a problem. We need to weigh the
> highly unlikely specter of that against the very real needs that the
> whitelist is trying to address.
>

You keep making this argument, but it's not based on a valid assumption.
That we haven't named and shamed is because we don't believe it's necessary
for the purposes of this discussion. It's a real problem, it happens more
often than is acceptable, and there's literally no evidence to support your
assumption that whitelisted add-ons are less likely to have problems. If
anything, whitelisted add-ons are going to be the primary target for
acquisition and attack once signing is mandatory, because they have the
best chance of evading detection.

-- Mike

[Brian Nakamoto]

Hi Jorge,

It would be helpful to know approximately how many add-ons would be affected by each proposed condition. Also, can you reiterate the goal(s)? e.g. Whose add-ons is AMO trying to fast-track?

I know there are companies who would be willing to pay AMO for faster add-on reviews. Part of the revenue from such a program could be used to grant free or discounted expedited service to organizations or individuals who can't afford to pay.

Thanks!

-Brian

[Jorge Villalobos]

Hello Brian,

On 11/18/15 3:19 PM, Brian Nakamoto wrote:
> Hi Jorge,
>
> It would be helpful to know approximately how many add-ons would be affected by each proposed condition.

There might be a couple hundred add-ons with over 100K active users. The
other conditions are more difficult to measure, but if we were to
implement whitelisting, I would expect only a couple dozen or so add-ons
to be on that list. More than that would be really hard to keep track of
and would increase the existing risks.

> Also, can you reiterate the goal(s)? e.g. Whose add-ons is AMO trying to fast-track?

That's a very good question. For me, the goal would be to fast-track
add-ons that:

* We know are extremely unlikely to cause problems in the near future.
* Would always require a manual review under the current system.
* Have release timing restrictions that give them very little time
between submitting to AMO and distributing to users, and have a large
user base awaiting these updates.

> I know there are companies who would be willing to pay AMO for faster add-on reviews. Part of the revenue from such a program could be used to grant free or discounted expedited service to organizations or individuals who can't afford to pay.

This has come up many times in the past, and our response has always
been that we don't want to create a layered system that is unfair for
the developers who can't pay. I think that's still true.

Also, it's always been on us to create a system that is fast enough, and
resource it appropriately (with capable reviewers). For the past years
we have failed in this area, but we're finally addressing it. We're
ramping up the amount of paid, near-full-time reviewers who can handle
the more difficult reviews, and we're also growing our volunteer team.
Lack of money isn't really the problem :).

> Thanks!
>
> -Brian
>

Jorge

[Brian Nakamoto]

Hi Jorge,

Whitelisting an add-on that you know is extremely unlikely to cause a problem in the near future comes with inherent risk when the add-on is modified (i.e. updated). However, I don't have any ideas at the moment for pre-release risk mitigation that's any more efficient than a manual review if one was required in the first place...

Perhaps you could enlist the help of each add-on's 100K+ user base. Limit the release of an update to a subset of users, and solicit them for simple feedback: Does this add-on work as expected? Chrome routinely asks users of Web Store extensions for feedback.

Example:
1. Release to 1% of the installed base; collect feedback for 12 hours.
2. Release to 10% of the installed base; collect feedback for another 12 hours.
3. Release to 50% of the installed base; collect feedback for another 12 hours.
4. and so forth...

- New users wouldn't be able to install the updated version until the update has been made available to the entire installed base.
- Ideally, the sub-samples would be sufficiently diverse in terms of Firefox version, OS, geo, etc.
- I think Firefox updates work in a similar manner (sans prompting users for feedback)?

Thanks!

-Brian

[Dan Stillman]

No, Jorge just said he didn't know of any examples of legitimate
front-loaded unlisted extensions that later became malicious — not that
he didn't want to name them. All we're asking for is hard evidence that
we can use to evaluate the current whitelist rules to see if we can
allay these concerns. Details matter — otherwise it's just fearmongering.

It seems like you're forgetting that whitelisted unlisted extensions are
the status quo, going back many years. If no one can point to
significant problems with front-loaded unlisted extensions — and no one
has, in any discussion I've seen going back a year and a half — then
that's quite relevant in terms of weighing the risks against the very
real consequences of no whitelisting (e.g., no Zotero).

Your argument that front-loaded whitelisted add-ons will become a
primary target once signing is mandatory makes no sense, either,
because, again, whitelisting is the status quo! Nothing about AMO
extensions is changing. Right now, if you want to purchase or attack an
extension, you can choose an AMO extension or an unhosted one. It seems
people are choosing AMO extensions. So either the purchasers/attackers
are really dumb or there's something intrinsic about extensions that are
developed by identifiable developers and hosted from known websites that
makes them less vulnerable.

Yes, any current level of purchases of or attacks on unhosted extensions
(which, again, no one can point to) could conceivably become focused on
the whitelisted extensions under the new system, but that's why the
whitelist will have an even higher bar than the status quo, limited to a
small, carefully vetted group of conscientious, dedicated developers
with specific needs and known identities that already have very strong
incentives to avoid purchase/attack. If your choice is to try to 1) buy
out Zotero, 2) buy out a single developer with a popular AMO extension
and spend a few hours crafting some code that will make it past an
overworked AMO reviewer, or 3) buy out a single developer with a popular
unlisted extension and spend a few minutes crafting some code that will
make it past automated signing (or do something that's not even
disallowed for web installs, like add ad injection with a buried
opt-out), you're not going to choose Zotero.

[Emiliano Heyns]

On Wednesday, November 18, 2015 at 8:43:24 PM UTC+1, Dan Stillman wrote:

> This metric isn't our fight, and I'm actually not sure if Emiliano is
> even having trouble passing automated review at the moment,

I can pass automated review currently because of two reasons:

* I have cut features and rejected feature requests that I thought were interesting because I would have to suffer manual review if I included them.
* I heavily re-use Zotero internals so I shift the problem there.

I also had to spend time rewriting things to code that is entirely the same but where one formulation triggers a rejection and the other does not. I think this is asinine (especially because the passing formulation is less readable) but whatever.

I would prefer not to have to rely on any of these by necessity, but what you gonna do? I have resigned myself to the idea that I can only offer that which auto-signing allows because anything else would mean I'd have to violate the trust I have built up with my users, who know I am not perfect as a software engineer but that I provide fast fixes. This process absolutely requires me to do debug/instrumented builds for debugging. Even 24/7 one-hour review times would put an unbearable burden on that process.

BTW, even the whitelisting doesn't address the need for instrumented builds. I'm certain I saw the idea of a chrome-style developer mode floated somewhere -- I hope that that will come to fruition.

On Wednesday, November 18, 2015 at 11:24:51 PM UTC+1, Jorge Villalobos wrote:

> There might be a couple hundred add-ons with over 100K active users. The
> other conditions are more difficult to measure, but if we were to
> implement whitelisting, I would expect only a couple dozen or so add-ons
> to be on that list. More than that would be really hard to keep track of
> and would increase the existing risks.

...

> > I know there are companies who would be willing to pay AMO for faster add-on reviews. Part of the revenue from such a program could be used to grant free or discounted expedited service to organizations or individuals who can't afford to pay.
>
> This has come up many times in the past, and our response has always
> been that we don't want to create a layered system that is unfair for
> the developers who can't pay. I think that's still true.

So instead we create a layered system that is unfair for developers who address a niche. I have no idea what kinds of cost would be involved, but I'd much rather pay than wait. I've seen no principled argument to justify the one kind of unfairness over the other.

Look, I know it's going to happen, and I know it will be a pain for me. I will have to come to terms with that. The constitution of my personality necessitates I react against what I perceive to be flawed reasoning. It doesn't mean I oppose the idea behind the signing.

Are there any new stats on the improved review times? The last one I saw was at https://blog.mozilla.org/addons/2015/11/04/add-ons-update-73/ and that still seems to celebrate "less than seven weeks" review times(assuming prelim review stands for full review in the case of unlisted extensions). Of course "less than seven weeks" could mean one hour, but it seems implausible that that is what is meant there.

[Dan Stillman]

So in the interest of moving forward a bit more concretely:

On 11/18/15 8:33 PM, Dan Stillman wrote:
> If your choice is to try to 1) buy out Zotero, 2) buy out a single
> developer with a popular AMO extension and spend a few hours crafting
> some code that will make it past an overworked AMO reviewer, or 3) buy
> out a single developer with a popular unlisted extension and spend a
> few minutes crafting some code that will make it past automated
> signing (or do something that's not even disallowed for web installs,
> like add ad injection with a buried opt-out), you're not going to

> choose Zotero [or, at least, you're not going to succeed].

I would hope that everyone can at least agree with the above statement.
Maybe you can quibble with "hours"/"minutes", but I'm assuming everyone
is realistic enough to acknowledge that bypassing a reviewer or the
scanner (or just adding allowed opt-out ad injection) isn't that
difficult for someone committed to doing so, and is certainly easier
than paying George Mason University to sell Zotero for malware and also
paying off the tenured professors from various universities that sit on
the board of the non-profit corporation that owns the Zotero trademark
and also paying off individual Zotero developers so that they keep
providing hourly support in the forums despite the permanent costs to
their own reputations so that no one realizes anything is up, all while
keeping the sale a secret so it doesn't become a massive story in the
academic/open-source world that AMO editors would quickly hear about.

Assuming we can agree on that, if you're convinced that there will
(suddenly) be a rash of attempts to buy out other whitelisted
front-loaded unlisted extensions, what specific additional conditions
would you add to the proposal that would allow you to slot in other
extension names for "Zotero" in my quoted statement above?

Or can we just agree that the proposed whitelist rules (preferably with
my proposed swapping of difficulty-passing-automatic-review for ADU), on
top of the nature and history of front-loaded unlisted extensions, will
already make that statement sufficiently likely to be true for any
whitelisted extension?

[Gervase Markham]

On 17/11/15 22:55, Jorge Villalobos wrote:
> already, so I'm starting a new thread so we can focus on this topic. The
> proposal is that we create a mechanism where AMO auto-approves and signs
> add-ons under certain circumstances.

Just to say: a massive thanks for listening and for taking up this idea :-)

Gerv

[Jorge Villalobos]

Yes, Firefox releases are handled that way. However, doing this for
multiple add-ons would require too much development time and staff
monitoring the feedback for it to work. Developers have asked us in the
past for ways to do staggered releases, so maybe we will add this
feature eventually. I don't think we will for this particular case, though.

Jorge

[Jorge Villalobos]

On 11/18/15 1:42 PM, Dan Stillman wrote:
> OK, but just to be clear, then still no one can provide an example of a
> front-loaded, unlisted extension that was legitimate and then became
> illegitimate.

Absence of evidence is not evidence of absence. One the main reasons
we're implementing this system for is that the non-AMO world was a huge
black hole.

> I'm not denying that there might have been some, or that
> it's possible, but I don't think we can draw any conclusion other than
> that it's not a problem that merits significant concern — and that
> whitelisted extensions would be even less likely to be a problem.

I don't think that's a safe conclusion to draw at all. Certainly not the
conclusion that best ensures our users' safety.

> We
> need to weigh the highly unlikely specter of that against the very real
> needs that the whitelist is trying to address.

The whitelist is meant to benefit a very small group of developers, and
I don't think its merits need to be balanced with that of the system as
a whole and its inherent risks. It's meant to be an exception, not a norm.

[Jorge Villalobos]

Most add-ons are being reviewed within a day or two. Those that don't
are the most complex ones and can take between a few days to several
weeks. The 7 weeks numbers are based on the state of the queues and not
the time it takes individual submissions to be reviewed. So, that number
is dependent on a large review backlog that needs to be handled by admin
reviewers. The admin review team has grown in the past few weeks, so
that is being addressed, albeit slowly. I would expect that the queues
will be back down to acceptable levels (all reviews done in less than a
week) within a month, but I can't give any guarantees.

Jorge

[Dan Stillman]

On 11/24/15 5:17 PM, Jorge Villalobos wrote:
> On 11/18/15 1:42 PM, Dan Stillman wrote:

>> I'm not denying that there might have been some, or that
>> it's possible, but I don't think we can draw any conclusion other than
>> that it's not a problem that merits significant concern — and that
>> whitelisted extensions would be even less likely to be a problem.

> I don't think that's a safe conclusion to draw at all. Certainly not the
> conclusion that best ensures our users' safety.

Right, so I grew tired of claims about "users' safety":

http://danstillman.com/2015/11/23/firefox-extension-scanning-is-security-theater

Abridged version on firefox-dev:

https://mail.mozilla.org/pipermail/firefox-dev/2015-November/003554.html

You can respond to that if you like. Every argument you've made in this
discussion to date is void in light of those examples.

(But yes, even in the hypothetical world you've created where scanning
does anything, a whitelisted extension would be less less likely to be a
problem, simply by virtue of whitelisted extensions being more clearly
identified. Federal prison provides a pretty strong disincentive against
criminality.)