Re: Owner for Commit Access Policy

[Gregory Szorc]

On Wed, Aug 3, 2016 at 8:20 PM, Mitchell Baker <mitc...@mozilla.com> wrote:

> ideal followup is governance ... cross posting to reach those likely to be
> interested
>
> I'm currently the owner of the Commit Access Policy module. That's
> because I wrote the original policy and did what was necessary to get it
> implemented. (That's old history!) I was also engaged in the rewrite to
> the current policy but not at the same level. There's a separate module for
> implementation, owned by Marcia Knous.
>
> Someone closer to our code should own this policy going forward. I have a
> few ideas but there are many people who have become active whom I don't yet
> know. So if there's someone you think should own this policy please do let
> me know. It should be someone familiar with how things work, who has a
> sense for good, workable practices that protect are code and a good
> communicator.
>
> current policy is here:
> https://www.mozilla.org/en-US/about/governance/policies/commit/access-policy/
>

I'm going to say something that might be a bit contentious: I think a
single commit access policy for all of Mozilla reflects the needs of
Mozilla from several years ago, not the needs of Mozilla today. The world
has changed. Mozilla has changed. The policy was written before distributed
version control was popular, before services like GitHub.

The reality of today is that the "Mozilla Commit Access Policy" is
effectively the "Firefox Commit Access Policy." There are large Mozilla
projects to which the existing policy does not apply or is simply ignored.
On GitHub, each organization or project has the freedom to define its own
policy. This is both good and bad. Mostly good for the flexibility and
convenience. Bad in that it has historically been the wild west and there
are some gotchas in GitHub's permissions model that can lead to access
being granted where it shouldn't be. See https://wiki.mozilla.org/Github
for more on the policy that more or less governs the "mozilla" organization
on GitHub. Of course there are other Mozilla-affiliated organizations, like
Rust, Servo, Bugzilla, TaskCluster, ...

I'm not sure how formal we want to be on a commit policy that attempts to
govern all of Mozilla and/or that governs less established projects or
projects outside the Firefox umbrella.

I do believe that Firefox needs a formal policy. I consider security and
legal requirements as significant drivers of at least the Firefox commit
policy. So having someone with well-formed connections to those groups and
who knows the server maintainers would be ideal. I know Doug Turner has
expressed interest in the Firefox commit policy and his org runs the
Mozilla-hosted version control servers and Firefox automation. Ditto for
Lawrence Mandel and Jonathan Griffin. Hal Wine has done a lot of work on
establishing sanity to GitHub access, has familiarity with Firefox access
and security concerns, has access to the version control servers, and seems
to have connections throughout Mozilla. Those are the first names that jump
out to me. But I think choosing someone really depends on the scope of the
module/policy going forward...

[Mitchell Baker]

Ah yes, good point. For me personally, there's very little that's
contentious about adapting our policies to reflect changing engineering
practices. When I wrote a bunch of our original policies many years
back, they were a reflection of actual practice at the time with our
aspirations. So updating policy to reflect today's practices seems a
mark of a well managed project. So probably there's a bunch to do here.

On the scope side, does "firefox" capture it? If we mean the set of
code that ships as part of Firefox, then it clearly includes Gecko,etc,
which we would want.

I agree that acting as if this policy governs code it doesn't is not
helpful. And by being clear what it does cover, it also makes it clear
where we are making a different risk / convenience / reward balance.

Mitchell

[Gervase Markham]

On 04/08/16 06:06, Gregory Szorc wrote:
> I'm going to say something that might be a bit contentious: I think a
> single commit access policy for all of Mozilla reflects the needs of
> Mozilla from several years ago, not the needs of Mozilla today. The world
> has changed. Mozilla has changed. The policy was written before distributed
> version control was popular, before services like GitHub.

I don't think that's contentious; I think it's a totally accurate
assessment :-)

> The reality of today is that the "Mozilla Commit Access Policy" is
> effectively the "Firefox Commit Access Policy."

Yep. And we should probably rename it as such.

> I'm not sure how formal we want to be on a commit policy that attempts to
> govern all of Mozilla and/or that governs less established projects or
> projects outside the Firefox umbrella.

I had a few abortive goes at this a few years ago; it's an enormous
effort to get everyone on the same bandwagon, and just leads to the
creation of bureaucracy for no value. Let's not try it again. But I
agree with you that having a formal policy for Firefox, and any repos
which are upstream of it, makes sense. Knowing who can check in to a
codebase which gets shipped to hundreds of millions of people is a good
idea.

Gerv

[Gervase Markham]

On 04/08/16 16:22, Hal Wine wrote:

> On Thu, Aug 4, 2016 at 1:48 AM, Gervase Markham <ge...@mozilla.org
> <mailto:ge...@mozilla.org>> wrote:
>
> I had a few abortive goes at this a few years ago; it's an enormous
> effort to get everyone on the same bandwagon, and just leads to the
> creation of bureaucracy for no value. Let's not try it again.
>

> Actually, there are some modern attempts at this - times have changed.

I'm not sure any of the things you name amount to an attempt to write a
single policy for all Mozilla repos governing who can check in or not; I
accept that there have been more scope-limited efforts, of course.

> But I
> agree with you that having a formal policy for Firefox, and any repos
> which are upstream of it, makes sense. Knowing who can check in to a
> codebase which gets shipped to hundreds of millions of people is a good
> idea.
>

> Since key upstream repos are now on GitHub (e.g Rust), this really means
> we need a plan that covers GitHub, imo.

A very fair point. (Although it need not cover all of our Github.)

> As is often the case, these "nice to haves" are "underfunded mandates"
> until something happens. There are a few of us who keep trying to push
> the rock up the hill in between the events that get everyone's attention.

:-) It's one of those "buying insurance" things - if we don't do this,
perhaps nothing bad will happen, but perhaps something will, and it'll
be much worse for not having done it.

Gerv

[Gregory Szorc]

On Wed, Aug 3, 2016 at 10:31 PM, Mitchell Baker <mitc...@mozilla.com>
wrote:

> Ah yes, good point. For me personally, there's very little that's
> contentious about adapting our policies to reflect changing engineering
> practices. When I wrote a bunch of our original policies many years back,
> they were a reflection of actual practice at the time with our

> aspirations. So updating policy to reflect today's practices seems a marof

> a well managed project. So probably there's a bunch to do here.
>
> On the scope side, does "firefox" capture it? If we mean the set of code
> that ships as part of Firefox, then it clearly includes Gecko,etc, which we
> would want.
>

I want to say yes. However, even defining "code that ships as part of
Firefox" under the same policy could be difficult.

We've always had 3rd party code like libbzip2 and more recently libraries
like ICU and WebRTC vendored in mozilla-central. These were projects where
Mozilla didn't really do much upstream work or at least weren't in the
driver's seat: we just periodically dumped upstream changes into
mozilla-central.

With the Oxidation of Gecko, we're starting to vendor Rust components that
are primarily developed by Mozilla but are done so outside the Firefox
umbrella. There are even tentative plans to have Servo "mirrored" into
Firefox repositories in real time, effectively bringing the Servo project
into scope for Firefox's policies (in theory).

With Firefox's Go Faster project, we are starting to ship code to users
that never lands in an existing Firefox repository on hg.mozilla.org. It
doesn't necessarily even go through the same code review workflow or tools.
This includes the entirety of the Test Pilot "experiments," for example
(see https://github.com/mozilla/testpilot).

There are projects like Firefox for iOS that use the "Firefox" brand name
but live outside the traditional Firefox code/tools umbrella.

If you want to be cynical or paranoid, you could say these things are
threats to the best practices and policies that have been established
around Firefox [for Android/Desktop + Gecko]. They definitely test the
boundaries of what a "Firefox Commit Access Policy" means.

I can make the argument that there should be a governance module covering
the rules for what gets shipped to Firefox [for Android/Desktop] users (not
necessarily limited to the mozilla-central bits) and how that is reviewed,
audited, etc. The commit access policy would be under purview of that
module.

FWIW, I've been put in a few positions over the years as a Firefox build
system peer where I was asked to review code that introduced 3rd party
source into Firefox. Oftentimes I was the one asking questions about
install size bloat, license review, etc. I did this because they are
important questions to ask, not because I was following some procedure (no
well-defined or followed procedure exists AFAIK). (I've wondered what would
have happened if they didn't land on my plate and nobody asked these
questions.) I can also point to instances where large Firefox features were
developed at Mozilla outside of Firefox's more stringent peer review
workflows and then were effectively thrown over a wall and incorporated
into Firefox. Firefox Sync was more or less done this way. There are things
in the Firefox Sync code that would have never received review from a
Firefox module peer. But by the time we wanted to "graduate" the "Weave"
extension to a core feature of Firefox, it was either spend months to
rewrite things or ship. We chose to ship and we're still living with the
technical debt and performance issues as a result.

I'm no fan of bureaucracy and excessive policy, but given the security,
privacy, stability, performance, and long-term engineering productivity
implications of shipping code to hundreds of millions of users and
maintaining that code, I think I would feel better if there were more
formal governance for incorporating/shipping code into/with Firefox not
developed under the traditional Firefox module governance system. This
would include decisions/policies on not only 3rd party code like WebRTC and
ICU, but also Mozilla-influenced projects like Rust and Servo components
and Test Pilot. As it stands, there is no central entity fulfilling that
role (I think the closest we have is Release Management and they tend to
not get involved until close to ship time).

> I agree that acting as if this policy governs code it doesn't is not
> helpful. And by being clear what it does cover, it also makes it clear
> where we are making a different risk / convenience / reward balance.
>
> Mitchell
>
>
>
>
> On 8/3/16 10:06 PM, Gregory Szorc wrote:
>
> On Wed, Aug 3, 2016 at 8:20 PM, Mitchell Baker <mitc...@mozilla.com>

> wrote:
>
>> ideal followup is governance ... cross posting to reach those likely to
>> be interested
>>
>> I'm currently the owner of the Commit Access Policy module. That's
>> because I wrote the original policy and did what was necessary to get it
>> implemented. (That's old history!) I was also engaged in the rewrite to
>> the current policy but not at the same level. There's a separate module for
>> implementation, owned by Marcia Knous.
>>
>> Someone closer to our code should own this policy going forward. I have
>> a few ideas but there are many people who have become active whom I don't
>> yet know. So if there's someone you think should own this policy please do
>> let me know. It should be someone familiar with how things work, who has a
>> sense for good, workable practices that protect are code and a good
>> communicator.
>>
>> current policy is here: https://www.mozilla.org/en-US/
>> about/governance/policies/commit/access-policy/
>>
>

> I'm going to say something that might be a bit contentious: I think a
> single commit access policy for all of Mozilla reflects the needs of
> Mozilla from several years ago, not the needs of Mozilla today. The world
> has changed. Mozilla has changed. The policy was written before distributed
> version control was popular, before services like GitHub.
>

> The reality of today is that the "Mozilla Commit Access Policy" is

> effectively the "Firefox Commit Access Policy." There are large Mozilla
> projects to which the existing policy does not apply or is simply ignored.
> On GitHub, each organization or project has the freedom to define its own
> policy. This is both good and bad. Mostly good for the flexibility and
> convenience. Bad in that it has historically been the wild west and there
> are some gotchas in GitHub's permissions model that can lead to access
> being granted where it shouldn't be. See https://wiki.mozilla.org/Github
> for more on the policy that more or less governs the "mozilla" organization
> on GitHub. Of course there are other Mozilla-affiliated organizations, like
> Rust, Servo, Bugzilla, TaskCluster, ...
>

> I'm not sure how formal we want to be on a commit policy that attempts to
> govern all of Mozilla and/or that governs less established projects or
> projects outside the Firefox umbrella.
>

[Mike Connor]

I think there are at least five reasonably distinct classes of repositories
in active use. I'm sure we could be more fine-grained as needed. I would
suggest that if someone were to draft a new policy, access to each class of
repo should have a separate sign-off process.

* Firefox and upstream components (across Hg and Github)
* Product localization repositories (mostly strings)
* Key web infrastructure and tooling (i.e. Mozilla services, www.mozilla.org,
etc)
* Tools and libraries that we and others build and use.
* Experiments and side projects (add-ons, proof of concept services, even
try server, etc)

Ultimately, this is about trust, not technical competency. In practice we
don't vouch for commit access based on technical skills/reviewer ability,
we do it because we believe we can trust that individual to a) behave
intelligently in dealing with repos and b) maintain necessary security over
key access to prevent system compromises. What we're ideally constructing
here is a policy to scope and manage trust domains for accessing Mozilla
repositories.

In that light, I think the path to victory is:

* Identify the right groupings of repositories (as above, or different)
* Identify the right verification process for getting access to each
grouping
* Implementing necessary technical changes (including potentially splitting
or reworking the Mozilla Github org) to ensure sane group-based controls
are in place.

While I'm in agreement with some of the concerns Greg's just raised around
integration from third parties, I think that's outside of the scope of
commit access policy, and everything to do with whether the current module
ownership structure for our products is effective. That's a separate,
complicated discussion worth having.

-- Mike

On Thu, Aug 4, 2016 at 4:48 AM, Gervase Markham <ge...@mozilla.org> wrote:

> On 04/08/16 06:06, Gregory Szorc wrote:

> > I'm going to say something that might be a bit contentious: I think a
> > single commit access policy for all of Mozilla reflects the needs of
> > Mozilla from several years ago, not the needs of Mozilla today. The world
> > has changed. Mozilla has changed. The policy was written before
> distributed
> > version control was popular, before services like GitHub.
>

> I don't think that's contentious; I think it's a totally accurate
> assessment :-)
>

> > The reality of today is that the "Mozilla Commit Access Policy" is
> > effectively the "Firefox Commit Access Policy."
>

> Yep. And we should probably rename it as such.
>

> > I'm not sure how formal we want to be on a commit policy that attempts to
> > govern all of Mozilla and/or that governs less established projects or
> > projects outside the Firefox umbrella.
>

> I had a few abortive goes at this a few years ago; it's an enormous
> effort to get everyone on the same bandwagon, and just leads to the

> creation of bureaucracy for no value. Let's not try it again. But I

> agree with you that having a formal policy for Firefox, and any repos
> which are upstream of it, makes sense. Knowing who can check in to a
> codebase which gets shipped to hundreds of millions of people is a good
> idea.
>

> Gerv
>
> _______________________________________________
> firefox-dev mailing list
> firef...@mozilla.org
> https://mail.mozilla.org/listinfo/firefox-dev
>

[Axel Hecht]

FWIW, for the 300+ accounts that I vouched for, I relied on risk
management. I looked at what I knew about the folks that I vouched for,
and what they're intending to do, and what mitigation strategy there was
when things wouldn't go as intended.

That risk management also shaped how we're shipping localizations of
Firefox today. That we only ship revisions that went through an
independent technical review for branded builds is part of that.

I wouldn't use the word "trust" to describe what I've done for VCS
accounts so far.

Axel

[Axel Hecht]

FWIW, for the 300+ accounts that I vouched for, I relied on risk
management. I looked at what I knew about the folks that I vouched for,
and what they're intending to do, and what mitigation strategy there was
when things wouldn't go as intended.

That risk management also shaped how we're shipping localizations of
Firefox today. That we only ship revisions that went through an
independent technical review for branded builds is part of that.

I wouldn't use the word "trust" to describe what I've done for VCS
accounts so far.

Axel

On 04/08/16 19:58, Mike Connor wrote:

[Gervase Markham]

On 04/08/16 18:46, Gregory Szorc wrote:
> I want to say yes. However, even defining "code that ships as part of
> Firefox" under the same policy could be difficult.

You are right. However, you make a good case for why it's a worthy
endeavour.

> We've always had 3rd party code like libbzip2 and more recently libraries
> like ICU and WebRTC vendored in mozilla-central. These were projects where
> Mozilla didn't really do much upstream work or at least weren't in the
> driver's seat: we just periodically dumped upstream changes into
> mozilla-central.

I'm not sure we can do much about that; getting Mozilla people to ramp
up on each of these codebases sufficiently to review the incoming
changes for maliciously-introduced security bugs would be a massive
effort. :-|

> I can make the argument that there should be a governance module covering
> the rules for what gets shipped to Firefox [for Android/Desktop] users (not
> necessarily limited to the mozilla-central bits) and how that is reviewed,
> audited, etc. The commit access policy would be under purview of that
> module.

That sounds like a wise idea. The person in charge would effectively be
the "Firefox Architect" - responsible for the big picture about how we
go about building the features defined by the product team.

Gerv

[Gervase Markham]

On 04/08/16 18:46, Gregory Szorc wrote:

> I want to say yes. However, even defining "code that ships as part of
> Firefox" under the same policy could be difficult.

You are right. However, you make a good case for why it's a worthy
endeavour.

> We've always had 3rd party code like libbzip2 and more recently libraries
> like ICU and WebRTC vendored in mozilla-central. These were projects where
> Mozilla didn't really do much upstream work or at least weren't in the
> driver's seat: we just periodically dumped upstream changes into
> mozilla-central.

I'm not sure we can do much about that; getting Mozilla people to ramp
up on each of these codebases sufficiently to review the incoming
changes for maliciously-introduced security bugs would be a massive
effort. :-|

> I can make the argument that there should be a governance module covering
> the rules for what gets shipped to Firefox [for Android/Desktop] users (not
> necessarily limited to the mozilla-central bits) and how that is reviewed,
> audited, etc. The commit access policy would be under purview of that
> module.

[Gervase Markham]

On 04/08/16 18:46, Gregory Szorc wrote:

> I want to say yes. However, even defining "code that ships as part of
> Firefox" under the same policy could be difficult.

You are right. However, you make a good case for why it's a worthy
endeavour.

> We've always had 3rd party code like libbzip2 and more recently libraries
> like ICU and WebRTC vendored in mozilla-central. These were projects where
> Mozilla didn't really do much upstream work or at least weren't in the
> driver's seat: we just periodically dumped upstream changes into
> mozilla-central.

I'm not sure we can do much about that; getting Mozilla people to ramp
up on each of these codebases sufficiently to review the incoming
changes for maliciously-introduced security bugs would be a massive
effort. :-|

> I can make the argument that there should be a governance module covering
> the rules for what gets shipped to Firefox [for Android/Desktop] users (not
> necessarily limited to the mozilla-central bits) and how that is reviewed,
> audited, etc. The commit access policy would be under purview of that
> module.