Howdy Everyone,

As discussed at the Gecko 1.9 meeting we are reviving the trunk
sheriffs.  People have already starting their duty as detailed on the
schedule here: http://wiki.mozilla.org/Sheriff_Schedule.  If you can't
make your time please just edit and sign up for a different day.

All the info on what a sheriff does:            
        http://wiki.mozilla.org/Sheriff_Duty

Regression Policy:
        http://www.mozilla.org/hacking/regression-policy.html

Other useful stuff:
        http://www.mozilla.org/hacking/working-with-seamonkey.html

We can discuss at the Gecko 1.9 meeting tomorrow if anyone has any
questions.

Best,

Schrep

Sounds good to me.

That's the PERFORMANCE regression policy.
What is the FUNCTIONAL regression policy?

I've been wondering about this for some time.

Over the last year or two, we've had several enhancement checkins that
caused significant numbers of regressions, quite a few of which STILL
are not fixed, some of which are crashers.

One RFE caused something like 33 regression bugs to be filed, of which
about 20 are still unresolved.  I keep wondering why some sheriff didn't
(and doesn't) back that out.

Of course, some of them have been in for SO LONG now, that it's kinda
too late for a sheriff to take corrective action now.  But then one
wonders why the regressions are allowed to live for SO long.

Maybe this question belongs in m.governance?

Have you asked the sherriff to do so, or at least brought it to their
attention. Ultimately module owners are responsible for the code in their
modules, and you can appeal to them if there is a question.

Could you be more specific, please?

--BDS

I don't know which one(s) Nelson is thinking about,
but I would suggest
Darin's [ Bug 326273 – Implement nsIThreadManager ]
as an example.

Not questionning the improvement(s) made by this checkin;
but wondering about the various long time regressions:
tab drag'n'drop indicator, tree not scrolling, (flash/js) memory leak, ...

Yes, that's the one that triggered my question.  There are many others
also, each less severe, but taken together they are quite a reduction
in usability.

... numerous password prompt issues, including a crash

See the whole list (for that bug) at
https://bugzilla.mozilla.org/showdependencytree.cgi?id=326273&hide_re...

Note that some of the regressions are shown as blockers of that bug, and
others as dependencies of it.

But my question really is about policy.  IMO, the trunk has been going
steadily down hill over the past 18 months, with new regressions
appearing that never seem to get fixed.  I don't understand why this is
being tolerated.  Seems like there's no pressure to fix regressions.
I wish there was a policy that created such pressure.

(Does this belong in .governance ? )

Nelson Bolyard schrieb:

The trunk is more stable than it ever was in the time when I found
developments in the Mozilla community most exiting, i.e. back in the
days when Mozilla didn't even have a version number, when we went with
milestones instead.

The trunk is a bleeding-edge development beast, it's not intended to be
production quality - if it would, our product would be as current, new
and exciting as Netscape 4.75 when it was released.

Regressions are what new, big code changes cause all the time, with no
chance to reasonably avoid them. And in most cases, those code changes
fix more problems than they regress. Some regressions are not found
during private testing and not during review and peer testing, they are
only found when some wider audience is testing that stuff (e.g. the
current problem of SeaMonkey tab titles not showing up due to a Core
change, as Firefox tabbrowser works correctly). Some regressions are
known even before checkin, but it's more helpful and less work to land
the patch and fixing the regressions afterwards, as long as trunk stay
dogfood-able.

This is good and reasonable development strategy. In closed source,
trunk would be the in-house bleeding-edge development dump (those
companies often don't even review all of that). As we are open source,
everyone can access and test that code, and find and file the
regressions, so that they get fixed over time.

Only after all the needed feature work for a new release (say FF3 in
that case) has been landed, there will be a stabilizing phase with a
feature freeze some thing resembling it, where regression-fixing becomes
the primary target and Betas are made. That doesn't mean regression
fixing is unimportant right now, but it's not the main target of
development atm. The stabilizing phase for FF3 has not yet begun. And
it's good this way.

Anyone who can not tolerate regressions should never even think of
touching trunk.

At least that's my opinion regarding that topic.

Robert Kaiser

Yes, the building process works well,
but the legacy features are not that stable.

We understand that regressions (can) happen,
the "complain" is about such regressions that affect (daily) Trunk testing:
*Memory leak from Darin's patch: forces to close and restart app
"regularly", which prevent "long time" testing ... for months.
*Blank tabs from Jonas's patch: very annoying for daily users ... but
this case is different as it seems it will be fixed within days.

Not too sure that it is so good that way:
in the past, there have often been complains about reporting/blocking
bugs to late in the cycle: these or those that these may be hidding;
moreover that means digging into code that was "forgotten" for a long time.

In another thread, sorting of (Core/FF3) blocking bugs has now started,
and target release selected: this is (now) good news, as you say.
What makes us wonder is that it is "discovered" that there is a "huge"
number of such blockers/regressions...

I think we do tolerate that they happen:
it could even be said that catching them is the point of testing
nightlies, couldn't it;
the point here is how long they last after being reported...

And this is mine :->

That's really really wrong.  All regressions should be one or the other
for any given bug (preferably blockers of the bug, imo, but consistency
is more important than which exact direction is used).

Yes.  It has.  :(  This part of the reason why we now have over 300 bugs
that are blocking1.9+, and why we're re-instituting sheriffs, as far as
I can see.

-Boris

True, but the point is to fix them once that's happened.

Not quite true, but we're working on the avoidance methods (e.g testing).

While this may be true, the exact things that are regressed matter more
than how many of them are, in some ways.

Somewhere here you should have "in a timely manner".

This is one key point.  Our trunk is only sort of dogfood-able now, and
many times over the last 5-6 months it hasn't been dogfood-able at all.

That last conclusion doesn't necessarily follow.  To get them fixed you
need someone fixing them.

The problem with that approach is that the longer a regression is in the
tree, the harder it gets to fix (not to mention that it becomes very
hard to back out the patch if it becomes clear that the regression is
not fixable and the cure is worse than the disease).

Also, we need to be in this "stabilizing phase" as of a month of two ago
if we're aiming for a 2007 release, imo.  How long do you estimate it
will take us to fix 300+ nontrivial (else they would have been fixed
already) bugs?

There's a world of difference between "regressions" and "regressions
that never get fixed".

-Boris

Oh, I'm sure they fix problems that were inconveniencing the developer
who made them, but what about their impact on USERS?  The regressions
listed for bug 326273 are all broken user features.  Name one user
feature that was actually fixed or enhanced by that checkin.  It may
have been elegant, but it broke WAY too much to have been tolerated as
it was.

And there are mega-exceptions.  Checkins that cause multiple tens of
regressions, including crashes.  No way will anyone prove that that
checkin was a net positive.  It should have been backed out, or the
person who did it shuold work around the clock to fix the damage.
That's the missing policy, IMO.

If so, then it ought not to take 12 months or more to fix the regressions,
and the people who know about the regressions a priori' should not walk
away from the job until the regressions are fixed.

We're very unlikely to get volunteers to spent large amounts of effort,
rewriting formerly working code to get it to work again, after it was
broken by someone else's checkin.  This demotivates developers and drives
them away.  They think "why should I keep working on this when others can
break my code and I must pay for their mistakes?"  and "I worked hard to
get that working, and now person X has broken it.  Let HIM fix it."

And we want to avoid a culture in which some developers are allowed to
cause regressions without being forced to fix them, while other developers
are required to fix theirs.  That happened at Netscape and there was
nearly a revolt over it.

I fully expect that FF3 will ship with most of the regressions from bug
326273 still unfixed.  Many of them are over a year old now.  It's
obvious that no-one thinks it's his job to fix them.  I don't believe
that 2 months before FF3 releases a lot of developers will suddenly
realize that they should have been working on these things all along.

Also, when a regression becomes a year old, or more, it becomes necessary
to CONVINCE people that these ARE regressions.  People reason: "it's
worked this way for 6 months, therefore it's worked that way forever,
and it should not be changed now."

Exactly, regressions that last 3-4, even 6 weeks are one thing.
Regressions that last 17 months are quite another.

The problem is that about half the mozilla developers think that regressions
should be listed as blockers, and half think they should be dependents.

When a bug is resolved/fixed, no-one is clear on how to interpret blockers
of that bug that are still unfixed.  Evidently they weren't really blockers,
or else the bug should not have been marked fixed.  And a bug that was
caused by the "fix" for another bug is clearly not dependent on that fix.
If anything, it is dependent on the undoing of that fix.

I think we need new categories of links to other bugs, including
"Caused" and "caused by" for regressions, and "see-also" for related bugs
that are not causal nor blockers nor dependencies.

IIRC, it paved the way for network traffic during modal dialogs.
--
Blake Kaplan

Sure.  That's why I said we should pick one or the other for this bug
and just do it consistently (for this bug).   If someone has already
gone through and sorted out which bugs in those dep lists are
regressions, it would be great to put the results somewhere so we can
change the deps (assuming said person doesn't change the deps himself).

The way I tend to think of it, is that they're blockers for landing the
bug fix on a branch, say.  Or rather that they _should_ have been
blockers for the bug landing but were simply not discovered in time.
But again, I'm fine with either direction as long as we're consistent
for any given bug.

Absolutely.  There are existing bugs on Bugzilla for it.  :(

-Boris

Which is in fact what causes a lot of the regressions, because code that
was not expecting this broke.

But yes, that was in fact the main point of the change, and the main
user benefit -- now a modal dialog posed from one Firefox (or Seamonkey,
or Thunderbird) window doesn't prevent network access started from other
windows before the dialog came up from completing.

-Boris

The question about policy has been answered, we had a policy problem
back then, but we don't have one today. We have sheriffs back, we have
perf testing, feature testing, we have daily smoketests by QA. All of
which are things we didn't have a year ago.
It doesn't help fixing the regressions that darin got reassigned since
then, of course.

If you care about that bug, and I read you do, mind filing a bug "Fix
nsIThreadManager regressions" and make sure that all regressions do
block that bug? And then you can start to see which of those are
reproducable (threads make me ask that), and request blocking1.9 for
those bugs. At least for those that you don't want fx3 to ship with.

That's the only way to get things done here.

Axel

Boris Zbarsky schrieb:

I've been using 1.8 branch over that period, so I can't clearly tell -
I'll be able to tell more from personal experience once suiterunner
lands and I'll move to that for daily usage.
 From earlier experience, I've seen thought that it varies a lot what
different people call dogfood-able, as I've been using milestone builds
and nightlies in old times on a daily production basis when most
Netscape engineers considered the codebase non-dogfood-able...

Right. And I guess it's probably no good idea to keep code in when its
creator doesn't actively work on its regressions, esp. if they break
dogfood-ability.
Which reminds me that it might be a good idea to start using dogfood and
catfood keywords again in Bugzilla.

Sure, and we probably need to do a better job in fixing regressions. Or,
actually, those among us who cause them - I'm not writing a lot of code
myself, I'm just OKing temporary regressions made by others in SeaMonkey
currently to get suiterunner landed (which is nearly as dogfood-able
right now as xpfe trunk - with mailcompose crashing on both, they're
probably labeled "unusable" anyways until that regression is fixed...)

Well, this might be some difference between us: I'm not aiming for a
2007 release, as I'm in doubt if the app I'm planning for (SeaMonkey 2)
will be able to make that. I for myself would be quite happy to see
Gecko 1.9 slip to spring 2008 or such.
But then, I'm no Firefox or Gecko lead, and those are who decide on
schedules, I guess ;-)

Right. I meant to be talking about the former, which are hard to avoid
even if you're doing good testing (you might be able to reduce their
number though). The latter is surely something we surely don't want. (At
least if where are not intentional - as things like "unmaintained qt
port doesn't build any more" or "FF3 doesn't work on Win9x" are surely
regressions, but ones that are probably wanted and will not be fixed.)

Robert Kaiser

Nelson Bolyard schrieb:

I thought we had exactly that policy in place? I think that it's
probably only a matter of executing that policy.

Robert Kaiser

This, along with Boris' comment that "our trunk is only sort of
dogfood-able now" concerns me, as that's not the perception I have.

I wonder if these various regressions are time-bombs which are quietly
piling up, or if they're just the inevitable result of changes to a
complex product trading one set of bugs for another (albeit with a net
benefit, one hopes!).

I've been dogfooding trunk on OS X for quite a few months, and "grumpy
Schrep" has been strongly encouraging folks to be dogfooding trunk as
well. My general impression is that while there are certainly a number
of annoyances and quirks, it's entirely dogfoodable. I don't use FF2 at all.

I think it would be good to have a discussion (this discussion, I
suppose) on what the concerns are, and evaluate what needs to be done to
make sure the problem is scoped and on-track for resolution for Firefox 3.0.

Justin

I agree. If you are seeing bugs you would not want Firefox 3 to ship
with, please file them in bugzilla and notify the appropriate module
owners so they can be triaged.

I won't go so far as to say to nominate them as blocking-1.9/firefox-3,
because I'm unsure of exactly what the policy there is, but I think
everyone can agree that noises should be made about critical bugs that
need to be fixed before a release.

-Colin

It all depends on your task set and on your environment.

For example, on my machine (Linux) about one in three SVG testcases in
Bugzilla causes trunk Gecko to hang X; if I'm lucky I ssh in from
another machine quickly enough and kill the browser; 5-20 minutes later
(depending on how fast I moved) X stops hogging all the CPU and starts
reacting to input again.  If I'm not lucky, the computer stops
responding to the network and the only thing I can do is a hard reboot.

Now this clearly interferes with day to day work: if I make a layout or
XBL or content change and it causes an SVG regression (a normal
occurrence, really)... what am I supposed to do?  Load the testcase in
the bug and hope that I don't hit the above symptoms?  Ignore the bug?
Try to figure out what's going on by inspection of the testcase in a
text editor?  Upgrade my OS and hope a newer X versions deals better
with whatever we're throwing at it so I can continue working on trunk?

Fwiw, this last seems to be the official policy at the moment....  But
that doesn't change the fact that this is a dogfood regression for me.

Some of both; net benefit is hard to gauge because the cost-benefit
equation is so different for different people and in different situations.

Your text inputs must work more often than mine do, then.  I've not been
able to use trunk for more than an hour or two at a time on OS X the
last few times I tried; doing the "click in a textfield, alt-tab,
alt-tab" dance every minute is so distracting that it just makes it
impossible to get work done at anything resembling 100% productivity.

I feel that we need to:

1)  Nominate bugs for blocking Gecko 1.9 (or Firefox 3) as needed
2)  Triage said nominations, with regressions likely getting
     blocking status
3)  Find people to fix the blocking bugs.

Step 3 is hard.

-Boris

I would.  Given the number of blocking+ bugs already, and how many more
are likely to appear given the number of nominations we have, the
chances of something that is _not_ a blocker and not on the pet peeve
list of someone who goes and fixes it are slim, I suspect.

-Boris

Colin Barrett schrieb:

Should we go for using the dogfood (and maybe catfood) keywords in
Bugzilla again, and maybe do stats on them, trying to get them (at least
dogfood) down to zero even for nightlies and alphas?

Robert Kaiser

"Oh, Linux."

You've got a point there. At least, my impression is that Linux needs
the most help of the 3 main platforms.

Were you just using SVG on Linux as an example to hilight differences in
task set / environment, or have there been recent regressions (Cairo :-)
  that started killing X?

Absolutely.

Probably not! I end up doing the same kinds of workarounds as you
mention, but I just haven't personally found it to be something that
stops me from getting work done. I suppose this is the kind of
regression that falls into the "death by a thousand papercuts" category...

Perhaps we should track 'dogfood' bugs as a standard part of the weekly
FF3/Gecko meetings? Although, being more aggressive about backing out
new regressions is probably (but not always) a more efficient way to do
that for the future.

I also wonder if it might help to have longer terms for sheriffs? Say, 2
days to a week? That might help by having stronger ownership for backing
out checkins which don't immediately manifest themselves as a flaming
tinderbox inferno.

Justin

Both.  Clearly, if you're on Windows or OSX (or even just a newer X
version), whatever this particular regression is (and yes, it's fallout
from turning on cairo) is not dogfood for you.

It doesn't so much stop me as reduce the amount of work I can do.  Which
means I can more efficiently deal with my bug list if I don't use trunk
for my day to day browsing... which is bad, because then I'm not testing
the changes I make to it before checking them in. Which leads to more
regressions from those changes, usually.

The problem, as discussed, is defining "dogfood"....  I do think it's
worth tracking regression bugs.  It's hard to say which regressions are
1.9-only based on bugzilla, but we currently have 545 open bugs with the
"regression" keyword that were filed in the Core product 3 months after
1.8 branched or later (I used Nov 1, 2005 as the three months date).
433 of them were filed in the last 12 months (so since May 22, 2006).

The vast majority of these bugs are unowned (assigned to nobody@, etc).

Ideally this number would hit something close to 0 before we ship 1.9.
In my opinion.

-Boris

[...]

Have you tried logging in locally on a text console? (Ctrl-Alt-F2, go back to
X with Ctrl-Alt-F7). In my experience, even when X is hung, it's usually
possible to log in on /dev/tty and renice (or kill) the misbehaving Firefox
process. Then within a few _seconds_ X starts reacting to input again.

Best regards,
Tony.
--
If a jury in a criminal trial stays out for more than twenty-four
hours, it is certain to vote acquittal, save in those instances where
it votes guilty.
                -- Joseph C. Goulden