Regressions

[Shawn Wilsher]

How are we supposed to be handling regressions nowadays? We've been
pretty good about commenting in the bug about a regression, and saying
that that was done in this forum, but then we often seem to see no other
information about it. Should we just start backing things out when a
regression looks real, and go from there? Or something else?

Cheers,

Shawn

[Benjamin Smedberg]

Yeah, unless there is a positive response "I'm working on it" or something
like that we should be backing out more aggressively for perf regressions.
They are easy to let slide and then be difficult to back out later.

--BDS

[Ehsan Akhgari]

I totally agree.  I think over time we have become more lenient on performance regressions, which is not a good thing.  And performance regressions are difficult to track if not backed out soon, and we don't have a way of keeping track of them like we do for test failures (filing orange bugs.)  And the worst thing is that our regression policy [1] clearly states what needs to be done.  Has this policy changed since that document was written?

[1] http://www.mozilla.org/hacking/regression-policy.html

--
Ehsan
<http://ehsanakhgari.org/>
 

[Mike Beltzner]

On 2009-11-16, at 5:08 PM, Ehsan Akhgari wrote:

On Mon, Nov 16, 2009 at 4:55 PM, Benjamin Smedberg <benj...@smedbergs.us> wrote:

I totally agree.  I think over time we have become more lenient on performance regressions, which is not a good thing.  And performance regressions are difficult to track if not backed out soon, and we don't have a way of keeping track of them like we do for test failures (filing orange bugs.)  And the worst thing is that our regression policy [1] clearly states what needs to be done.  Has this policy changed since that document was written?

We talked about this at a development meeting, and the agreement was as Benjamin stated. I do what I can by identifying and commenting, but it's really up to the sheriff to police. Perhaps we need to keep a clearinghouse to track these things?

cheers,

mike

[Ehsan Akhgari]

On Mon, Nov 16, 2009 at 5:47 PM, Mike Beltzner <belt...@mozilla.com> wrote:

On 2009-11-16, at 5:08 PM, Ehsan Akhgari wrote:

On Mon, Nov 16, 2009 at 4:55 PM, Benjamin Smedberg <benj...@smedbergs.us> wrote:

I totally agree.  I think over time we have become more lenient on performance regressions, which is not a good thing.  And performance regressions are difficult to track if not backed out soon, and we don't have a way of keeping track of them like we do for test failures (filing orange bugs.)  And the worst thing is that our regression policy [1] clearly states what needs to be done.  Has this policy changed since that document was written?

We talked about this at a development meeting, and the agreement was as Benjamin stated. I do what I can by identifying and commenting, but it's really up to the sheriff to police. Perhaps we need to keep a clearinghouse to track these things?

I think it's best if we don't let these regressions to live in the tree for too long, because people may start basing off their work over these and they might get difficult to back out as time passes.  And I agree that sheriffs need to be strict about these regressions.

What exactly do you have in mind as a clearinghouse?

--
Ehsan
<http://ehsanakhgari.org/>
 

[Shawn Wilsher]

On 11/16/09 2:47 PM, Mike Beltzner wrote:
> We talked about this at a development meeting, and the agreement was as
> Benjamin stated. I do what I can by identifying and commenting, but it's
> really up to the sheriff to police. Perhaps we need to keep a
> clearinghouse to track these things?

I think it's probably best to just backout and let the owner of the bug
determine what the cause is. Right now, we don't really have it setup
such that the owner actually does an research to establish why we
regressed, and then fix it since their work stays in the tree. In a
perfect world, people would do this, but we sadly don't live in such a
place.

I'd be supportive of immediate backouts, and figure out what to do with
the patch after the fact.

Cheers,

Shawn

[Justin Dolske]

On 11/16/09 1:55 PM, Benjamin Smedberg wrote:

> Yeah, unless there is a positive response "I'm working on it" or something
> like that we should be backing out more aggressively for perf regressions.
> They are easy to let slide and then be difficult to back out later.

This. Along with the caveat that we tread lightly where the "regression"
is not clear and obvious... We still have lots of cases where the
automated regression notices are false alarms, or have questionable
regression ranges.

Justin

[Mike Beltzner]

On 11/17/2009 6:31 PM, Justin Dolske wrote:
> This. Along with the caveat that we tread lightly where the "regression"
> is not clear and obvious... We still have lots of cases where the
> automated regression notices are false alarms, or have questionable
> regression ranges.

And those are the ones we can lose track of; which is why I suggested
the clearinghouse. Basically when we cross sheriff boundaries or a
weekend, we need better recordkeeping.

cheers,
mike