New Policy: Marking Bugzilla bugs for features riding behind a pref

[Emma Humphries]

Hello,

We control enabling many features and changes to Firefox using preferences.

Program and Release management as well as PI need a better view of this.

We've written a new policy which you can read on our nascent bug-handling
mini-site:

https://github.com/mozilla/bug-handling/blob/master/policy/feature-flags.md

To summarize, when you are releasing a feature that "rides behind a flag",
on the bug for the feature:

* set the behind-pref flag to +
* set the qa-verify flag to ?
* note the bug in the Firefox Feature Trello board

We expect qa-verify to be set to + before enabling prefs on features.

We'll be going over this at two upcoming meetings, with Reese's and JoeH's
teams.

There are two, known open questions to resolve on the policy:

* Features developed over multiple releases with individual patches not
verifiable by external testing (passing unit tests, but not integration
tests) such as Hello and WebRTC
* Features are often turned on in Nightly, ignoring prefs using compiler
directives, and kept off in Beta and Release. Is this the right thing to
do, or should we be flipping prefs from on to off when going from Nightly
to Beta and forwards?

The bug handling documentation is a GitHub repo to enable web publishing,
please follow up there, using Issues and PRs, rather than to this email.

I want to acknowledge and thank: Liz Henry, Ritu Kothari, Resse Morris,
Erin Lancaster, Ryan VM, Thomas Elin, and Adam Roach for their comments and
feedback on this.

Thank you!

-- Emma Humphries

[Nicholas Alexander]

On Wed, May 2, 2018 at 4:57 PM, Emma Humphries <em...@mozilla.com> wrote:

> Hello,
>
> We control enabling many features and changes to Firefox using preferences.
>
> Program and Release management as well as PI need a better view of this.
>
> We've written a new policy which you can read on our nascent bug-handling
> mini-site:
>
> https://github.com/mozilla/bug-handling/blob/master/
> policy/feature-flags.md
>
> To summarize, when you are releasing a feature that "rides behind a flag",
> on the bug for the feature:
>
> * set the behind-pref flag to +
> * set the qa-verify flag to ?
> * note the bug in the Firefox Feature Trello board
>
> We expect qa-verify to be set to + before enabling prefs on features.
>
> We'll be going over this at two upcoming meetings, with Reese's and JoeH's
> teams.
>
> There are two, known open questions to resolve on the policy:
>
> * Features developed over multiple releases with individual patches not
> verifiable by external testing (passing unit tests, but not integration
> tests) such as Hello and WebRTC
> * Features are often turned on in Nightly, ignoring prefs using compiler
> directives, and kept off in Beta and Release. Is this the right thing to
> do, or should we be flipping prefs from on to off when going from Nightly
> to Beta and forwards?
>

Not all features are feasible to ship behind feature flags. Fennec
features that interact with the OS directly, in particular, can sometimes
just be all or nothing; and I would anticipate that things that interact
directly with newer App stores (iOS features, say, or Windows Store
features in the future) will become more common. We can't have a policy
that fights against that trend.

Nick

[Adam Roach]

On 5/3/18 12:18 PM, Nicholas Alexander wrote:
> Not all features are feasible to ship behind feature flags.

I'm pretty sure the proposed policy isn't intended to change anything
regarding features that ship without associated feature flags, nor is it
trying to get more features to ship behind flags than currently do. It's
just trying to rationalize a single, more managable process for those
that *do* ship behind flags.

/a

[Robert Helmer]

I agree that not every single feature is appropriate to ship behind a
feature flag, since the cost to maintain parallel implementations can
be huge, and has implications on things like the size of the
application and updates. In addition, if you look at
e10s/stylo/webrender we've set up parallel testing infrastructure,
which needs to be maintained for quite a while even after we've
enabled the feature.

We did however consider the benefits to be worth the complexity for
even the very large and complex projects mentioned above, and there
are many downsides and risks to not using a phased roll-out approach
(which can be done without feature flags), so I'd be interested in
continuing this discussion in a separate thread.

For this specific proposal, the only change I'd suggest is that we
should move away from the term "Pref Flip" in favor of "Feature Flag",
since (as evidenced in this thread so far) the latter is the more
standard industry term.

I also think there's an argument to be made that the pref system on
Firefox Desktop is not the right system for implementing feature flags
in general, but I'll leave that for a separate thread as well.

[Jonathan Kew]

On 03/05/2018 00:57, Emma Humphries wrote:

> To summarize, when you are releasing a feature that "rides behind a
> flag", on the bug for the feature:
>
> * set the behind-pref flag to +
> * set the qa-verify flag to ?
> * note the bug in the Firefox Feature Trello board
>
> We expect qa-verify to be set to + before enabling prefs on features.

I don't see a flag "qa-verify" in bugzilla; was this possibly a typo for
"qe-verify"?

Thanks,

JK

[Jared Wein]

I have a bug that will introduce a pref/feature-flag, but that is just one
bug that blocks a meta bug. The meta bug covers the whole feature.

Should these flags get set on the meta bug or on the bug that introduces
the pref? Testing the specific bug that introduces the pref won't provide
much coverage, but setting the flag on the meta bug may be confusing since
there will be no patches landed on that bug.

I also agree that updating the Trello tracking board is inconvenient (and I
also don't know where to find it). Could we just have a Bugzilla query for
this information?

Thanks,
Jared

> _______________________________________________
> firefox-dev mailing list
> firef...@mozilla.org
> https://mail.mozilla.org/listinfo/firefox-dev
>

[Emma Humphries]

My thanks to everyone who had commented on this so far. I’m replying to
your questions and comments in this email so that we don’t have a threading
catastrophe.

First, several of you mentioned the typo where I wrote qa-verify when the
flag is qe-verify. The policy document has the correct flag name.

Second, this is not process for the sake of process, but so that Program
and Release Management, PI, and engineering leadership can quickly assess
what is happening with features.

Please note that Program / Release Management did not come up with this
policy on our own. The Firefox engineering teams were simultaneously
looking to create a similar policy to further alleviate releases where a
feature is inadvertently not tested or enabled. A recent item was the
Retained Display list not be turned on.

There’s also some question about what qe-verify means. It can be requested
by anyone with editbugs and granted by anyone with editbugs. And `+` does
not mean the bug is verified, but that QA will verify the the fix and set
the bug’s resolution to VERIFIED. I’ve updated that in the document (at
https://github.com/mozilla/bug-handling/commit/9b461160df743262aa6ecb7e6a033872ce393ff1
).

Liz Henry points out we should be clear in the policy what VERIFIED means
here for testing and sign-off.

:mossop asked about the Trello board, it’s at
https://trello.com/b/8k1hT2vh/firefox (if you need access, please ask
:elan) and we use the board to track features of note so that we know to
communicate them through release notes and other channels. Since we’re
using a flag to indicate features behind a pref, we can use queries to
indicate features that need to go on that board.

Nick Alexander mentioned that features which interact with the OS, mobile
OSes (Android and iOS) are all or nothing. This policy does not mean you
must release behind a pref. But if you are releasing a feature that depends
on a change to your OS, the bug for the feature must note that. That sort
of change may not be atomic in nature, so early warning and notification of
Project and Release Management, and PI is important.

Jared Wein asked in the thread; MattN, and Mark Banner asked in Issues (
https://github.com/mozilla/bug-handling/issues/4, and
https://github.com/mozilla/bug-handling/issues/4) about feature work
involving large numbers of bugs associated with a [meta] bug. Mark’s
suggestion where the flag is associated with the meta bug, but not the
dependent bugs is most likely the way to go.

Ritu Kothari and D. Baron asked about managing features across release
trains. The behind-pref may need to be a status flag instead of a simple
flag since the target version field only indicates when a bug landed in M-C
and bugs supporting a feature may land across multiple releases. I’ve
started a discussion of this at
https://github.com/mozilla/bug-handling/issues/5.

Several of you note that we’ve overloaded preferences. And perhaps we
should think about releasing features under a feature flag system. I think
that would afford us a way to move features toggling to the pref panes
(even if we keep feature flags in a namespaced area in prefs.)

Finally, I need to talk more with Shield/Normandy about this and make sure
I’m not sabotaging you.

Thank you all for your feedback and for helping us ship high quality code.

— Emma

On Wed, May 2, 2018 at 4:57 PM Emma Humphries <em...@mozilla.com> wrote:

> Hello,
>
> We control enabling many features and changes to Firefox using preferences.
>
> Program and Release management as well as PI need a better view of this.
>
> We've written a new policy which you can read on our nascent bug-handling
> mini-site:
>
> https://github.com/mozilla/bug-handling/blob/master/policy/feature-flags.md
>

> To summarize, when you are releasing a feature that "rides behind a flag",
> on the bug for the feature:
>
> * set the behind-pref flag to +
> * set the qa-verify flag to ?
> * note the bug in the Firefox Feature Trello board
>
> We expect qa-verify to be set to + before enabling prefs on features.
>

> We'll be going over this at two upcoming meetings, with Reese's and JoeH's
> teams.
>
> There are two, known open questions to resolve on the policy:
>
> * Features developed over multiple releases with individual patches not
> verifiable by external testing (passing unit tests, but not integration
> tests) such as Hello and WebRTC
> * Features are often turned on in Nightly, ignoring prefs using compiler
> directives, and kept off in Beta and Release. Is this the right thing to
> do, or should we be flipping prefs from on to off when going from Nightly
> to Beta and forwards?
>