Updated Performance Regression Policy

[Joel Maher]

It has come to my attention that the policy we use for handling performance of mozilla-central (and related trees) is outdated:
http://www.mozilla.org/hacking/regression-policy.html

I have spent some time looking at what we do and could reasonably be doing with our current tools. The result is a newly proposed policy:
https://etherpad.mozilla.org/perf-policy

Please reply to this thread if there are questions, concerns or praises about this new proposed policy. Assuming all changes are voiced and added in the policy before November 15th, I will work on making this the official policy the week of November 18th.

[Gavin Sharp]

Can you give a brief summary of the changes?

Gavin

> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning

[Stephen Pohl]

Joel Maher wrote:
> It has come to my attention that the policy we use for handling performance of mozilla-central (and related trees) is outdated:
> http://www.mozilla.org/hacking/regression-policy.html
>
> I have spent some time looking at what we do and could reasonably be doing with our current tools. The result is a newly proposed policy:
> https://etherpad.mozilla.org/perf-policy
>

The new policy mentions that " There will be a few instances where there
is a performance regression and we approve this. These types of
regressions are not common and will require documentation as to why we
are regressing performance." Would it be possible to be specific about
who's to make the call? The original document says to "start a
discussion with dri...@mozilla.org". Is this still true? What kind of
documentation should be written to explain a performance regression? I'm
asking because this could come in very handy for bug 860493, which turns
on history swipe animations on OSX by default (bug 678392).

Thanks!

-Stephen

[Nicholas Nethercote]

On Tue, Nov 5, 2013 at 5:50 AM, Joel Maher <joel....@gmail.com> wrote:
>
> I have spent some time looking at what we do and could reasonably be doing with our current tools. The result is a newly proposed policy:
> https://etherpad.mozilla.org/perf-policy

"If a regression is posted to dev.tree-management and has no action
taken for 2 business days, Mozilla will start backing code out to
reduce the regression."

What happens in the (common) case where there's a regression and there
are a dozen patches with half a dozen authors in the regression
window? I received such an email just this morning, and I concluded
that the regression -- assuming it was real and not just noise --
almost certainly wasn't my fault. Though it's possible I was wrong...

Nick

[Joel Maher]

There is a scenario with our current tools which reports a regression no branch X, then when that merges into mozilla-central or mozilla-inbound, we see that regression again. I believe this is the case that you are seeing- a merge causes a regression alert.

Let me modify the policy to account for merges, we should have identified the regression earlier on and have a bug on file or a discussion surrounding that.

[Joel Maher]

> The new policy mentions that " There will be a few instances where there
>
> is a performance regression and we approve this. These types of
>
> regressions are not common and will require documentation as to why we
>
> are regressing performance." Would it be possible to be specific about
>
> who's to make the call? The original document says to "start a
>

> discussion with driv...@mozilla.org". Is this still true? What kind of

>
> documentation should be written to explain a performance regression? I'm
>
> asking because this could come in very handy for bug 860493, which turns
>
> on history swipe animations on OSX by default (bug 678392).
>
>
>
> Thanks!
>
>
>
> -Stephen

Thanks for calling this out and being proactive about changes you are making. Let me find a channel for notification of upcoming perf issues and a place to document this. Most likely this will be a distribution list to email and a wiki that journals changes and dates landed. A possible whiteboard tag to add to bugs as well.

-Joel

[Andrew McCreight]

----- Original Message -----
> On Tue, Nov 5, 2013 at 5:50 AM, Joel Maher <joel....@gmail.com> wrote:
> >

> > I have spent some time looking at what we do and could reasonably be doing
> > with our current tools. The result is a newly proposed policy:
> > https://etherpad.mozilla.org/perf-policy
>

> "If a regression is posted to dev.tree-management and has no action
> taken for 2 business days, Mozilla will start backing code out to
> reduce the regression."
>
> What happens in the (common) case where there's a regression and there
> are a dozen patches with half a dozen authors in the regression
> window? I received such an email just this morning, and I concluded
> that the regression -- assuming it was real and not just noise --
> almost certainly wasn't my fault. Though it's possible I was wrong...

Also, the bit about "Mozilla will start backing out" is weird. Who is the "Mozilla" actually responsible for doing this? The patch writer? The sheriffs? Module owners? Somebody else?

Andrew

>
> Nick

[Lawrence Mandel]

I added a comments section to the etherpad in order to track the comments in one place. I also added a number of comments to the pad.

Lawrence