Let's use feature switches (Re: new features, bitrot & killswitches)

[Tim Guan-tin Chien]

Hi,

TL;DR: Let's use feature flags and invest in our ability to use
feature flags while maintaining engineering quality.

== Problem statements and values ==

The original thread was very insightful. The essential conflict of
values can be stated as:

1. Ability to incorporate community (or, "volunteer labor") into
Firefox OS (a libre but also a client software project)
2. Ability to accommodate features driven by different stakeholders
(MoCo product team, or a specific volunteer/engineer) and on different
targets

v.s.

A. Ability to maintain engineering quality (test coverage etc.)
B. Ability to maintain a single product vision and uniformity.

and the proposed directions are

i. get more UX/product people to figure out keeping (B) while
accommodating (1) and (2)!
ii. Use feature switches, build flags without sacrificing (A) and to
fulfill (1) and (2).
iii. Add-ons to hack everything out-of-the-tree and out-of-our-minds.

and the problem raised on these directions are

i*: For (i), we can never hire that many people.
ii*: For (ii), we are going to sacrificing (A) at the end anyway.
iii* : For (iii), these add-ons will suffer the same problems with
traditional XUL add-ons. It can't do everything either.

== IMHO ==

Frankly I agree all of the statements above. So I tend to think which
of the goals we value more *and* which of the lesser values we could
bear when they give ways to the higher goal.

With that I will pick (1) and (2) over (A) and (B), and I propose (ii)
as the more attainable solution.

Why?

I think (i*) is a real issue. Even if we have the money, we can never
run any bigger organization effectively than we already are. I also
don't think it is in any way possible as a way to run a free software
project which the # of stakeholders mentioned in (2) is way more than
the one person named Steve Jobs.

I think (iii*) is real too, and I also think ask outside contributors
leave the upstream along and have fun with themselves is not a
responsible attitude to run a free software project. It would not be
the most inviting project that I want to devote my free time to.

(ii*) is real, however, it is the only problem (compare to (i*) and
(iii*)) that can be partly prevented through technical means.

We could have more automation in place to assert the quality of the
code base (but, yes, first we would have to fix the debts in our
automation...). We could make build script more sane when working with
these switches (no, existing front-end build tools don't address this
use case). We should have some place for runtime switches than
Developer menu. There are some many thing to do technically and last
time I checked we are mostly engineers.

The other reason I am in favor of (ii) is also because we want to move
to train model, so even features from the MoCo product management
would have to be hidden behind a flag too, and ideally, only ships
when it's ready. The works required to attend (1) and (2) would be
align to the company goal and our salaries and bonuses are justified,
even if the company only cares about (A) and (B).

This is in essence the never ending struggle between making a client
software v.s. running a free software project. 10 years ago Firefox
Desktop picked (iii) and lived with (iii*), and have tried to solve
(iii*) ever since. The struggle can also be mapped to MoCo management
v.s. Mozilla engineering too since the former are in charge of running
the product and the latter and supposedly responsible of running the
project.

Obviously if we decided to go (ii), we would really need to work hard
preventing (ii*). I would imagine the existing module ownership can
help too in keeping the code quality since that's what reviewers are
asked too.

Process-wise, that would also means UI changes can be checked-in
without UI review as long as the flag is not turned on for release.
Engineers would need to be careful to find the balance between
ignoring UX, land first and fix later for release, or waiting for the
UX spec before landing.

We however need a clearer mandate and priority on ensuring our ability
to use feature switches. Some technical detail would need to be sort
out too. I vaguely remember Jonas was working on this under the
mandate of getting train model running, but I don't know the progress
yet.

We would need a |about:config| like page for run-time switches
(Settings > Developers does not scale, trust me).
We would need to collect problem with our current build time flags
(For one, not being able to run integration tests on a feature if it's
not included in the build).

Tell me what you think and tell me the problems.

(if you are still reading ... well, thank you!)


Tim

On Fri, Sep 11, 2015 at 11:23 AM, Jim Porter <jpo...@mozilla.com> wrote:
> On 09/10/2015 07:45 PM, Dietrich Ayala wrote:
>> You've described an ideal version of a typical software development
>> cycle. (Or not ideal nor typical, depending on where we've all worked
>> before!)
>>
>> But we're not in that situation.
>>
>> We're in a high-stakes gambit against entrenched incumbents who can
>> out-hire us 1000 to 1 for far longer than we can attempt to keep up.
>
> I don't think we're any smarter than Apple or Google people. If they
> could produce software at the quality level they expect with fewer
> workers, they would. There isn't some shortcut we can take to make our
> labor more efficient than theirs. If we try to go faster, we'll
> sacrifice quality (possibly in worse UX or less code
> maintainability/testability).
>
> Given the choice, I'd rather have a feature-light phone that works
> really well than a feature-rich one with lots of bugs and bad UX.
>
>> We cannot compete symmetrically against the incumbents using their
>> methods. We must optimize for our unique strengths: Maximize
>> participation vectors and remove as many barriers to experimentation as
>> possible, to enable innovation to occur at scale.
>
> I assume this means something like "make it easy to incorporate
> volunteer labor into Firefox OS". In that case, I think it's even more
> essential to have a big UX team that can respond quickly; I really don't
> trust random volunteers to have a solid understanding of our UX goals.
> Granted, this could mean empowering some of the more visually-oriented
> developers with limited UX-powers, but that's not really much different
> from replacing some developers with UXers.
>
>> I'm asking about ways to accelerate feature development in ways that
>> allow more people to participate and more non-release experimentation to
>> occur in the project... and doing it in a way that minimizes the impact
>> on our product release cycle.
>
> Accelerating feature development in Firefox OS is probably the last
> thing I'm interested in. Firefox OS (especially master) is just too
> buggy for me. I'm happy using Firefox Nightly, since I've almost never
> been bitten by a serious bug there, but every time I try Firefox OS
> nightlies, some core feature is just completely broken. Perhaps it's not
> a fair comparison, since we don't have a gaia-inbound branch, but
> there's no reason that SMS should be broken on master, as you mentioned
> in another thread. (Incidentally, SMS being broken was why I gave up
> dogfooding the last time I tried several months ago.)
>
> - Jim
> _______________________________________________
> dev-gaia mailing list
> dev-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-gaia

[Andrew Sutherland]

I'm with Jim that feature switches are a bad idea, and that add-on's are
the best current and long-term option.

On Fri, Sep 11, 2015, at 12:40 AM, Tim Guan-tin Chien wrote:
> Tell me what you think and tell me the problems.

Complexity: the permutations of interacting features. The hardest thing
about software development is reasoning about what the software does.
Feature switches that interact with other features create exponentially
more complex situations.

Of course, not every feature interacts with every other feature, but
feature switches require a much higher level of discipline than an
explicit extension system. The safest implementations where features
live in their own files that are dynamically enabled/disabled ends up
looking like a nicely factored extension/add-on system.

The major risk with feature flags is that they let you get away with
cutting corners. In my time hacking on Thunderbird, I saw existing
preference-controlled features and newly proposed ones that seemed
trivial on their own; just a few lines of code and a pref. But these
greatly increased the potential number of ways things interacted.
Things would break and the provided patch in many cases was moving one
or two lines of code around with no tests and no comments that would
help ensure the code never moved again. Touching that code before tests
was added was stepping into a minefield of potential breakage and
pissed-off users. (:squib also hacked on Thunderbird, which is likely
we he is also so vehemently against feature switches.)

Add-ons are of course not a silver bullet, as your point iii* makes. If
the code an addon author is trying to influence doesn't have a clean
means of being influenced, we can encounter the hacks resorted to in
Firefox addons where Function.toSource() and regexp transforms could be
used or clobbering a function with a modified block of copied-and-pasted
code.

A great example of add-on based development is the atom text editor and
its package-centric architecture. Everything is a package, with
explicit consumed and provided services. This discipline and the
effectively enforced modular architecture allowing only explicit
coupling benefits both the core developers and the add-on developers.

And the good news is that we're already going in this direction. Teams
have had time to pay down technical debt. The Next Generation
Architecture (NGA) work is explicitly about modularizing things with
explicit contracts. The gaia-components and other web component efforts
are about modularizing and encapsulating our UI widgets and avoiding CSS
soup, etc.

The future's so bright we gotta wear Mozilla-branded hats because the
store doesn't sell shades[1].

Andrew

1: https://gear.mozilla.org/

PS: There are some proactive things we can do; I'm going to reply to
Dietrich's post directly since my reply here is already long.