WebCompat Go Faster Add-on

[Mike Taylor]

As discussed in the "Platform + Web Compat" session in London, the
webcompat team intends to build a Go Faster Add-on to give us the
ability to quickly respond to high profile site compatibility issues in
the coming quarter(s).

The intent is to be able deploy short-term fixes for high-impact,
user-facing problems while we work on medium- to long-term fixes (in
Gecko, or with partners and top sites), without needing to wait on trains.

I've written an explainer doc, with use cases and requirements here:

<https://wiki.mozilla.org/Compatibility/Go_Faster_Addon>

Comments welcome, thanks.

--
Mike Taylor
Web Compat, Mozilla

[Justin Dolske]

What's the policy for reviews and testing with this addon?

Justin

> _______________________________________________
> firefox-dev mailing list
> firef...@mozilla.org
> https://mail.mozilla.org/listinfo/firefox-dev
>

[Benjamin Smedberg]

On Mon, Jun 27, 2016 at 2:26 PM, Cory Price <cpr...@mozilla.com> wrote:

>
> On Mon, Jun 27, 2016 at 11:05 AM, Justin Dolske <dol...@mozilla.com>
> wrote:
>
>> What's the policy for reviews and testing with this addon?
>>
>

> You can see the current process for deploying things in the Go Faster
> documentation (Wiki <https://wiki.mozilla.org/Firefox/Go_Faster>, Release
> Process
> <https://docs.google.com/document/d/1x27I7hAmWDWiqk3o3YC3fklhE3N59bdgHCQHF5p_lkU/edit#>
> ).
>
> I've also added this to our system add-on pipeline
> <https://trello.com/b/moJCpVCD/go-faster-system-add-on-pipeline> which we
> discuss in our weekly meetings (invited Mike/Justin).
>

This seems like an inadequate answer to the particular question: who in
particular is the module owner of this code, who is responsible for doing
code review? And who is the QA/QE lead for this addon and what kind of
systems will be used to determine whether a particular webcompat tweak is
working both before before and after deployment?

--BDS

--
Benjamin Smedberg
Engineering Manager, Firefox

[Justin Dolske]

I'm not really clear what part of those docs covers my questions. Either
I'm missing it, or my question wasn't clear...

I'm asking because this is code shipping to end-users, so I'd expect it to
adhere to basically the same engineering standards as code that ships as
part of baseline Firefox. AFAIK this is the first time that's really been a
question -- our other system addons (Hello, Pocket) started off in Firefox,
and just changed their delivery vehicle.

To be more specific:

* Who are the people that will be reviewing code that ships in this addon?
I don't see anything in the docs about code review or who can do it. Will
there be a module? Would the relevant platform reviewers be used when
shimming in DOM APIs?

* The docs do expand a bit on testing, but it sounds like a one-time QA
signoff. That's an important part, but I don't see anything about automated
/ ongoing tests (against product code or website code). For example, if a
Gecko change breaks something the addon is relying on, when will that be
noticed? Or if the addon's patch for a site stops working (or, worse,
breaks it!) due to a change the site deploys after the addon is released,
when will that be noticed?

* Similarly to correctness testing, how is performance testing dealt with?

Justin

On Mon, Jun 27, 2016 at 12:09 PM, Cory Price <cpr...@mozilla.com> wrote:

> The answer was focused on the policy question which should be covered for
> the most part in the documentation. With respect to individual roles and
> responsibilities for this add-on, I've added it to the agenda in the
> aforementioned team check-in.
>
> On Mon, Jun 27, 2016 at 11:45 AM, Benjamin Smedberg <bsme...@mozilla.com

> --
> Cory Price
> /ckprice
>

[Mike Taylor]

> On Mon, Jun 27, 2016 at 2:26 PM, Cory Price <cpr...@mozilla.com
> <mailto:cpr...@mozilla.com>> wrote:
>
>
> On Mon, Jun 27, 2016 at 11:05 AM, Justin Dolske <dol...@mozilla.com
> <mailto:dol...@mozilla.com>> wrote:
>
> What's the policy for reviews and testing with this addon?
>
>
> You can see the current process for deploying things in the Go
> Faster documentation (Wiki
> <https://wiki.mozilla.org/Firefox/Go_Faster>, Release Process
> <https://docs.google.com/document/d/1x27I7hAmWDWiqk3o3YC3fklhE3N59bdgHCQHF5p_lkU/edit#>).

Thanks, I wasn't aware of the Release Process doc (so I guess I owe an
"Intent to Implement" email to the right lists in the coming weeks).

On 6/27/16 1:45 PM, Benjamin Smedberg wrote:
>
> This seems like an inadequate answer to the particular question: who in
> particular is the module owner of this code, who is responsible for
> doing code review?

How do other go faster addons operate? My naive answer is people on our
team.

> And who is the QA/QE lead for this addon and what
> kind of systems will be used to determine whether a particular webcompat
> tweak is working both before before and after deployment?

If you read the explainer doc, near the end we mention needing automated
testing to verify that patches are needed post-deployment. We've built
similar things in the past for testing "bugfix" regressions. The design
of that is TBD, but our team will build it and monitor it.

One of the requirements of this addon is that it can be (temporarily)
disabled so site owners can verify that they've fixed things. We don't
have dedicated QA resources, so this will likely be a manual process by
our team: turn it off, verify, turn it on, verify, etc.

[Mike Taylor]

On 6/27/16 3:20 PM, Justin Dolske wrote:
> I'm asking because this is code shipping to end-users, so I'd expect it
> to adhere to basically the same engineering standards as code that ships
> as part of baseline Firefox. AFAIK this is the first time that's really
> been a question -- our other system addons (Hello, Pocket) started off
> in Firefox, and just changed their delivery vehicle.
>
> To be more specific:
>
> * Who are the people that will be reviewing code that ships in this
> addon? I don't see anything in the docs about code review or who can do
> it. Will there be a module? Would the relevant platform reviewers be
> used when shimming in DOM APIs?

I don't know how the module system works wrt go faster add-ons. But
maybe someone can add some clarity around this this.

I'd likely be reviewing patches written against sites, within our team.
And we'd be asking people with more Firefox/XPCOM/go faster experience
to review the site patching infrastructure. We'd absolutely be asking
for review from DOM peers if and when we shim any APIs. If someone wants
to step up and volunteer to review all our patches, that's also great.

> * The docs do expand a bit on testing, but it sounds like a one-time QA
> signoff. That's an important part, but I don't see anything about
> automated / ongoing tests (against product code or website code). For
> example, if a Gecko change breaks something the addon is relying on,
> when will that be noticed? Or if the addon's patch for a site stops
> working (or, worse, breaks it!) due to a change the site deploys after
> the addon is released, when will that be noticed?

I sort of expanded on that in my reply to Benjamin. Right now it's a
very hand-wavey TODO item here:
<https://wiki.mozilla.org/Compatibility/Go_Faster_Addon#TODO>

We don't plan on shipping site patches without this kind of ongoing
sites -- sites change all the time, and often developers fix bugs
without letting us know (once we're at the outreach stage).

> * Similarly to correctness testing, how is performance testing dealt with?

For this go faster add-on, or go faster add-ons in general?

[Cory Price]

+gofaster@

On Mon, Jun 27, 2016 at 11:05 AM, Justin Dolske <dol...@mozilla.com> wrote:

> What's the policy for reviews and testing with this addon?
>

You can see the current process for deploying things in the Go Faster
documentation (Wiki <https://wiki.mozilla.org/Firefox/Go_Faster>, Release
Process
<https://docs.google.com/document/d/1x27I7hAmWDWiqk3o3YC3fklhE3N59bdgHCQHF5p_lkU/edit#>
).

I've also added this to our system add-on pipeline
<https://trello.com/b/moJCpVCD/go-faster-system-add-on-pipeline> which we
discuss in our weekly meetings (invited Mike/Justin).

> Justin
>
> On Mon, Jun 27, 2016 at 10:51 AM, Mike Taylor <mi...@mozilla.com> wrote:
>
>> As discussed in the "Platform + Web Compat" session in London, the
>> webcompat team intends to build a Go Faster Add-on to give us the ability
>> to quickly respond to high profile site compatibility issues in the coming
>> quarter(s).
>>
>> The intent is to be able deploy short-term fixes for high-impact,
>> user-facing problems while we work on medium- to long-term fixes (in Gecko,
>> or with partners and top sites), without needing to wait on trains.
>>
>> I've written an explainer doc, with use cases and requirements here:
>>
>> <https://wiki.mozilla.org/Compatibility/Go_Faster_Addon>
>>
>> Comments welcome, thanks.
>>

>> --
>> Mike Taylor
>> Web Compat, Mozilla

>> _______________________________________________
>> firefox-dev mailing list
>> firef...@mozilla.org
>> https://mail.mozilla.org/listinfo/firefox-dev
>>
>
>
> _______________________________________________
> firefox-dev mailing list
> firef...@mozilla.org
> https://mail.mozilla.org/listinfo/firefox-dev
>
>

--
Cory Price
/ckprice

[Cory Price]

The answer was focused on the policy question which should be covered for
the most part in the documentation. With respect to individual roles and
responsibilities for this add-on, I've added it to the agenda in the
aforementioned team check-in.

On Mon, Jun 27, 2016 at 11:45 AM, Benjamin Smedberg <bsme...@mozilla.com>
wrote:

>

> On Mon, Jun 27, 2016 at 2:26 PM, Cory Price <cpr...@mozilla.com> wrote:
>
>>
>> On Mon, Jun 27, 2016 at 11:05 AM, Justin Dolske <dol...@mozilla.com>
>> wrote:
>>
>>> What's the policy for reviews and testing with this addon?
>>>
>>
>> You can see the current process for deploying things in the Go Faster
>> documentation (Wiki <https://wiki.mozilla.org/Firefox/Go_Faster>, Release
>> Process
>> <https://docs.google.com/document/d/1x27I7hAmWDWiqk3o3YC3fklhE3N59bdgHCQHF5p_lkU/edit#>
>> ).
>>
>> I've also added this to our system add-on pipeline
>> <https://trello.com/b/moJCpVCD/go-faster-system-add-on-pipeline> which
>> we discuss in our weekly meetings (invited Mike/Justin).
>>
>

> This seems like an inadequate answer to the particular question: who in
> particular is the module owner of this code, who is responsible for doing

> code review? And who is the QA/QE lead for this addon and what kind of

> systems will be used to determine whether a particular webcompat tweak is
> working both before before and after deployment?
>

[Cory Price]

I met with Mike earlier today and answered a few of his mechanics questions
(with the help of Osmose)[0].

Timing for his add-on is end of Q3.

I've taken the action item to help coordinate testing around this add-on
over the next couple of weeks.

Thanks!

[0] https://pad.mocotoolsprod.net/p/cbs2hKntP9

> --
> Mike Taylor
> Web Compat, Mozilla
>

--
Cory Price
/ckprice