Shipping releases with known tracked bugs

[Ehsan Akhgari]

A while ago, bug 749010 was reported which was about a perma-orange on
our Linux64 mochitest-chrome suite due to a crash. This bug only
affected Firefox 13 which was in beta back then, and it was marked as
tracking-firefox13+ on 2012-04-30. Then people discussed whether to
back out bug 736028 which was apparently what triggered it, and that
didn't get approved, and bug 749010 was marked as
status-firefox13:wontfix (which was the wrong call IMO, more on that
below) on 2012-05-03, to which people objected and set the status back
to "affcted" on 2012-05-07. Then more discussion and debugging happened
on the bug, Nick posted a patch and Ben had a few nits over it, and then
nothing happened on the bug and we released Firefox 13 on 2012-06-05
with bug 749010 withstanding (and indeed if you look at
https://tbpl.mozilla.org/?tree=Mozilla-Release, you'll see that the
latest test runs on Linux64 PGO did experience crash.)

Now, we ship software with known crashes and bugs all the time, and
there is analysis in this bug saying that the crash did not affect
anybody in the wild, which I trust and I think the right call was
probably made in not getting bug 749010 backed out, especially since we
did have a fix for the crash at hand. But still, the fact remains that
this bug was a persistent crash inside our test suite, which means that
we had no test coverage on everything that was tested after the crashing
test. So we shipped a release without getting full test coverage on one
of our tier 1 platforms. This is the reason that I think marking bug
749010 as "wontfix" was the wrong call.

What bothers me a lot more though is how we went through a release with
a tracking+ status:affected bug and nobody noticed it. This query shows
all such bugs for Firefox 13 <http://bit.ly/Oqh75w> (with a total of 13
such bugs), and the same query for Firefox 14 <http://bit.ly/NfhKRN>
shows 28 bugs.

My impression had always been that we make sure to either fix things for
a release, or wontfix them, so I find it very strange that we have
shipped a release with known tracked items without having a final call
on whether we need to fix or wontfix them. I'm not 100% sure what the
process for making this call is, but as a developer I always relied on
the assumption that marking something as tracking+ will make sure that
it doesn't fall through the cracks, and seeing that this doesn't happen
in practice makes me very nervous.

For example, I think in case of bug 749010 at the very list we should
have disabled that test, so that we would only lose coverage on a single
test.

If this way of bugs falling through the cracks is not expected, we
should think about a way to fix it.

Cheers,
Ehsan

[Alex Keybl]

> My impression had always been that we make sure to either fix things for a release, or wontfix them, so I find it very strange that we have shipped a release with known tracked items without having a final call on whether we need to fix or wontfix them. I'm not 100% sure what the process for making this call is, but as a developer I always relied on the assumption that marking something as tracking+ will make sure that it doesn't fall through the cracks, and seeing that this doesn't happen in practice makes me very nervous.

There's little difference between a tracked bug being marked as affected or wontfix at time of release - it does not affect engineering process in any way. Either way, the bug is left unfixed at time of release. We review the full list of un-fixed bugs as we approach release, and make sure to cast a wider net than is necessary to ensure that things on the line stay on our radar.

Wontfixing the bug prior to the end of the cycle may not have been the right call, but we do at least partially rely on a sense of urgency with test-related issues from engineering, and didn't get the impression that a bug fix here was absolutely critical. We keep our eye on external feedback, and didn't know of any related fallout.

-Alex

> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning

[Ehsan Akhgari]

On 12-07-09 1:34 PM, Alex Keybl wrote:
>> My impression had always been that we make sure to either fix things for a release, or wontfix them, so I find it very strange that we have shipped a release with known tracked items without having a final call on whether we need to fix or wontfix them. I'm not 100% sure what the process for making this call is, but as a developer I always relied on the assumption that marking something as tracking+ will make sure that it doesn't fall through the cracks, and seeing that this doesn't happen in practice makes me very nervous.
>
> There's little difference between a tracked bug being marked as affected or wontfix at time of release - it does not affect engineering process in any way. Either way, the bug is left unfixed at time of release. We review the full list of un-fixed bugs as we approach release, and make sure to cast a wider net than is necessary to ensure that things on the line stay on our radar.
>
> Wontfixing the bug prior to the end of the cycle may not have been the right call, but we do at least partially rely on a sense of urgency with test-related issues from engineering, and didn't get the impression that a bug fix here was absolutely critical. We keep our eye on external feedback, and didn't know of any related fallout.

OK, that's true, but was this bug in particular triaged after the status
was set back to affected? At that time the bug did have enough
information stating that we're losing test coverage on it (see
https://bugzilla.mozilla.org/show_bug.cgi?id=749010#c26).

Ehsan

[Alex Keybl]

Yep - comments 34, 40, 43, and 58 all occurred during regular or out-of-band triage after being marked as affected again. You'll note that comment 58 in particular was our team trying to align with engineering on the criticality of a fix prior to release.

-Alex

[Ehsan Akhgari]

On 12-07-09 1:52 PM, Alex Keybl wrote:
> Yep - comments 34, 40, 43, and 58 all occurred during regular or out-of-band triage after being marked as affected again. You'll note that comment 58 in particular was our team trying to align with engineering on the criticality of a fix prior to release.

OK, then what's the process of evaluating whether a tracking+ affected
bug is OK to be ignored? I really don't think that this particular bug
was OK to be ignored, and I'm trying to figure out where things went
wrong in the hopes of making sure that we don't make the same mistake again.

Ehsan

[Alex Keybl]

Tracked bugs are bugs that should be investigated and gain clarity prior to release, but are not guaranteed to be fixed on that same timeline. Bugs are deemed hard blockers closer to release based upon data that we have along with engineering intuition, as opposed to data that we do not. There are no hard and fast rules when we do not know the affected population or general user effect.

If we as an engineering organization believe that all orange bugs need to be fixed prior to release, release management will be happy to follow up with engineering even more in the future. Up until now though, we've chosen to focus our limited time on known critical issues.

-Alex

[Ehsan Akhgari]

On 12-07-09 2:05 PM, Alex Keybl wrote:
> Tracked bugs are bugs that should be investigated and gain clarity prior to release, but are not guaranteed to be fixed on that same timeline. Bugs are deemed hard blockers closer to release based upon data that we have along with engineering intuition, as opposed to data that we do not. There are no hard and fast rules when we do not know the affected population or general user effect.

How are those hard blockers distinguished with other tracking+ bugs?
It's not clear to me what the list of hard blockers that we need to get
fixed before the release is. I was under the impression that it's equal
to the list of tracking+ affected bugs...

> If we as an engineering organization believe that all orange bugs need to be fixed prior to release, release management will be happy to follow up with engineering even more in the future. Up until now though, we've chosen to focus our limited time on known critical issues.

I am not suggesting that at all. In the case of that particular bug, I
think we should have disabled that test before going to build for the
release (I don't actually know how risky the patch on that bug has been
for beta). But the reason I started this thread was to know what
developers should do in order to mark a bug as "absolutely needs
attention before release."

Cheers,
Ehsan

[Alex Keybl]

>> Tracked bugs are bugs that should be investigated and gain clarity prior to release, but are not guaranteed to be fixed on that same timeline. Bugs are deemed hard blockers closer to release based upon data that we have along with engineering intuition, as opposed to data that we do not. There are no hard and fast rules when we do not know the affected population or general user effect.
>

> How are those hard blockers distinguished with other tracking+ bugs? It's not clear to me what the list of hard blockers that we need to get fixed before the release is. I was under the impression that it's equal to the list of tracking+ affected bugs…

If you've got a critical blocker for release, you'll get emails/IRC pings from our team and will likely be invited to channel meetings in the weeks coming up to release to discuss strategies of nailing down things like STR in QA or externally. We don't mark these bugs differently, but it's typically clear from comments and QA engagement that the bug is a high priority.

That's not to say that the other tracked bugs should be neglected - they should still take precedent over forward feature work until they are untracked.

> But the reason I started this thread was to know what developers should do in order to mark a bug as "absolutely needs attention before release."

Commenting in the bug to that end or emailing releas...@mozilla.com will always help increase urgency if the current prioritization doesn't feel right from the engineering perspective.

-Alex

[Ehsan Akhgari]

On 12-07-09 4:35 PM, Alex Keybl wrote:
>>> Tracked bugs are bugs that should be investigated and gain clarity prior to release, but are not guaranteed to be fixed on that same timeline. Bugs are deemed hard blockers closer to release based upon data that we have along with engineering intuition, as opposed to data that we do not. There are no hard and fast rules when we do not know the affected population or general user effect.
>>

>> How are those hard blockers distinguished with other tracking+ bugs? It's not clear to me what the list of hard blockers that we need to get fixed before the release is. I was under the impression that it's equal to the list of tracking+ affected bugs…
>
> If you've got a critical blocker for release, you'll get emails/IRC pings from our team and will likely be invited to channel meetings in the weeks coming up to release to discuss strategies of nailing down things like STR in QA or externally. We don't mark these bugs differently, but it's typically clear from comments and QA engagement that the bug is a high priority.

Yep, I've been receiving those. How are those bugs tracked? I mean,
how should I know by just reading a bug whether _somebody_ is getting
those emails or not?

> That's not to say that the other tracked bugs should be neglected - they should still take precedent over forward feature work until they are untracked.

OK. But I assume that this doesn't mean that the number of tracking+
bugs that are not either fixed, disabled, or wontfix should be 0 prior
to a realize, correct? In that case, my question is: what does it mean
for a bug to be on that list. Are those things that we're explicitly
not going to fix on a release? (In which case, why would they not be
wontfixes?)

>> But the reason I started this thread was to know what developers should do in order to mark a bug as "absolutely needs attention before release."
>
> Commenting in the bug to that end or emailing releas...@mozilla.com will always help increase urgency if the current prioritization doesn't feel right from the engineering perspective.

Yeah, I've been communicating with release drivers about things which
require extra attention for some reason in the past. But for usual bugs
that I deal with every day (let's say a crash for example) I usually
determine whether it's a regression on nightly/aurora/beta, and set the
tracking flags respectively, then forget about the bug until there's a
patch on it (assuming that I won't be the person fixing it myself).
I've always relied on doing this as a way of tracking regressions in a
particular release, and I figured that if the status of that bug on a
tracked branch is not set to fixed by the time we get close to a
release, somebody will complain. But it seems like this model did not
work with bug 749010, and I want to know whether that is expected and I
should switch to another way of tracking those types of bugs...

Ehsan

[Asa Dotzler]

On 7/9/2012 1:18 PM, Ehsan Akhgari wrote:
> On 12-07-09 2:05 PM, Alex Keybl wrote:
>> Tracked bugs are bugs that should be investigated and gain clarity
>> prior to release, but are not guaranteed to be fixed on that same
>> timeline. Bugs are deemed hard blockers closer to release based upon
>> data that we have along with engineering intuition, as opposed to data
>> that we do not. There are no hard and fast rules when we do not know
>> the affected population or general user effect.
>
> How are those hard blockers distinguished with other tracking+ bugs?
> It's not clear to me what the list of hard blockers that we need to get
> fixed before the release is. I was under the impression that it's equal
> to the list of tracking+ affected bugs...

I'm pretty sure that we don't intend to have a list of traditional
"blockers". Those went away with the train model. Bugs marked tracking+
are the ones we want to track, not necessary block on.

I trust that the release management team evaluates all of these
repeatedly as we get close to a release and those that would cause us to
not ship if not fixed have the right amount of attention.

>> If we as an engineering organization believe that all orange bugs need
>> to be fixed prior to release, release management will be happy to
>> follow up with engineering even more in the future. Up until now
>> though, we've chosen to focus our limited time on known critical issues.
>
> I am not suggesting that at all. In the case of that particular bug, I
> think we should have disabled that test before going to build for the
> release (I don't actually know how risky the patch on that bug has been
> for beta). But the reason I started this thread was to know what
> developers should do in order to mark a bug as "absolutely needs
> attention before release."

I'm pretty sure that we don't intend to have a list of traditional
"blockers". Those went away with the train model. Bugs marked tracking+
are the ones we want to track, not necessary block on.

I trust that the release management team evaluates all of these
repeatedly as we get close to a release and those that would cause us to
not ship if not fixed have the right amount of attention.

[Ehsan Akhgari]

On 12-07-09 6:59 PM, Asa Dotzler wrote:
> On 7/9/2012 1:18 PM, Ehsan Akhgari wrote:
>> On 12-07-09 2:05 PM, Alex Keybl wrote:
>>> Tracked bugs are bugs that should be investigated and gain clarity
>>> prior to release, but are not guaranteed to be fixed on that same
>>> timeline. Bugs are deemed hard blockers closer to release based upon
>>> data that we have along with engineering intuition, as opposed to data
>>> that we do not. There are no hard and fast rules when we do not know
>>> the affected population or general user effect.
>>
>> How are those hard blockers distinguished with other tracking+ bugs?
>> It's not clear to me what the list of hard blockers that we need to get
>> fixed before the release is. I was under the impression that it's equal
>> to the list of tracking+ affected bugs...
>
> I'm pretty sure that we don't intend to have a list of traditional
> "blockers". Those went away with the train model. Bugs marked tracking+
> are the ones we want to track, not necessary block on.

The only reason that I talked about "hard blockers" was because Alex
mentioned them. But yeah, I don't expect us to block a release on a
specific bug being resolved. I do however think that a call needs to be
made on every tracking+ bug on whether it needs to get fixed, disabled
(backed out) or wont-fixed.

> I trust that the release management team evaluates all of these
> repeatedly as we get close to a release and those that would cause us to
> not ship if not fixed have the right amount of attention.

I trust that they do too, but my question still remains. As a person
reading the bug, how should I tell what the final call has been on a
tracking+ affected bug?

Ehsan

[Alex Keybl]

> The only reason that I talked about "hard blockers" was because Alex mentioned them. But yeah, I don't expect us to block a release on a specific bug being resolved. I do however think that a call needs to be made on every tracking+ bug on whether it needs to get fixed, disabled (backed out) or wont-fixed.

Something akin to a "hard blocker" (which I used colloquially) would be a tracked bug that causes us to scramble QA/support, land speculative fixes, spin out-of-band betas at the last minute, and could potentially delay a release. Thankfully it hasn't come to delaying a release yet (although we've come close, see [1]). We don't like to make the call on delaying a release until we have as much data as possible and we've exhausted our options.

> I trust that they do too, but my question still remains. As a person reading the bug, how should I tell what the final call has been on a tracking+ affected bug?

All bugs that remain on the tracked list at the time of release are bugs that we intentionally did not block the release for, but may still consider fixing in a .1 release if possible (especially based upon post-release feedback). I'm not sure what explicitly wontfix'ing them accomplishes, and I don't think it really represents the state of those bugs. We review the full list continuously in the weeks approaching release, as Asa mentioned.

-Alex

[1] https://blog.mozilla.org/futurereleases/2012/03/12/update-on-firefox-release-timing/

On Jul 9, 2012, at 4:21 PM, Ehsan Akhgari wrote:

> On 12-07-09 6:59 PM, Asa Dotzler wrote:

>> On 7/9/2012 1:18 PM, Ehsan Akhgari wrote:
>>> On 12-07-09 2:05 PM, Alex Keybl wrote:
>>>> Tracked bugs are bugs that should be investigated and gain clarity
>>>> prior to release, but are not guaranteed to be fixed on that same
>>>> timeline. Bugs are deemed hard blockers closer to release based upon
>>>> data that we have along with engineering intuition, as opposed to data
>>>> that we do not. There are no hard and fast rules when we do not know
>>>> the affected population or general user effect.
>>>
>>> How are those hard blockers distinguished with other tracking+ bugs?
>>> It's not clear to me what the list of hard blockers that we need to get
>>> fixed before the release is. I was under the impression that it's equal
>>> to the list of tracking+ affected bugs...
>>
>> I'm pretty sure that we don't intend to have a list of traditional
>> "blockers". Those went away with the train model. Bugs marked tracking+
>> are the ones we want to track, not necessary block on.
>

> The only reason that I talked about "hard blockers" was because Alex mentioned them. But yeah, I don't expect us to block a release on a specific bug being resolved. I do however think that a call needs to be made on every tracking+ bug on whether it needs to get fixed, disabled (backed out) or wont-fixed.
>

>> I trust that the release management team evaluates all of these
>> repeatedly as we get close to a release and those that would cause us to
>> not ship if not fixed have the right amount of attention.
>

> I trust that they do too, but my question still remains. As a person reading the bug, how should I tell what the final call has been on a tracking+ affected bug?
>
> Ehsan

[Dao]

On 10.07.2012 01:49, Alex Keybl wrote:
> All bugs that remain on the tracked list at the time of release are bugs that we intentionally did not block the release for, but may still consider fixing in a .1 release if possible (especially based upon post-release feedback). I'm not sure what explicitly wontfix'ing them accomplishes,

It would have given Ehsan a chance to disagree.

[akeybl]

Assigning extra urgency when we build our final beta (or after) is too late in the process. It'd really help if engineering could email us or comment in the bug if they find an issue more critical than the attention it is getting.

-Alex

Dao <d...@design-noir.de> wrote:On 10.07.2012 01:49, Alex Keybl wrote:
> All bugs that remain on the tracked list at the time of release are bugs that we intentionally did not block the release for, but may still consider fixing in a .1 release if possible (especially based upon post-release feedback). I'm not sure what explicitly wontfix'ing them accomplishes,

It would have given Ehsan a chance to disagree.

[Kyle Huey]

On Tue, Jul 10, 2012 at 7:52 AM, akeybl <ake...@mozilla.com> wrote:

> Assigning extra urgency when we build our final beta (or after) is too
> late in the process. It'd really help if engineering could email us or
> comment in the bug if they find an issue more critical than the attention
> it is getting.
>
> -Alex
>

I think the point is that there's no way to tell how critical the issue is
perceived as from the bug, beyond the binary tracking+/-.

- Kyle

[Robert Kaiser]

Ehsan Akhgari schrieb:

> OK. But I assume that this doesn't mean that the number of tracking+
> bugs that are not either fixed, disabled, or wontfix should be 0 prior
> to a realize, correct?

IMHO, if it would be a "needs to be driven to 0 by some means" list,
then it would be a blocker list, not a tracking list. In how I see it,
"tracking" means "we should watch this one closely" and AFAIK that's
being done in e.g. the channel meetings, but also outside of that by
release managers. I know that Sheila usually also looks at the open
trackers that are crashes a few times when we come towards the endgame
of a release, we usually talk through those within the CrashKill team.
Other teams might do similar things.

> In that case, my question is: what does it mean
> for a bug to be on that list

That we will take a look at those closely before deciding if we can go
along and release with them or not. At least that's my take on it.

Robert Kaiser

[akeybl]

All tracked bugs take priority over other forward work, so again, I'm not sure what adding granularity here would accomplish.

Just so people understand where I'm coming from, it really feels like we're overrotating on something that isn't a major problem and can generally be resolved through conversations when there's concern. Tools and process don't take the place of a conversation saying "this bug really scares me".

-Alex

[Gavin Sharp]

On Tue, Jul 10, 2012 at 8:02 AM, akeybl <ake...@mozilla.com> wrote:
> All tracked bugs take priority over other forward work, so again, I'm not sure what adding granularity here would accomplish.

It would allow developers to avoid needing to monitor the tracking
lists closely, and instead rely on their bugmail to make sure they
don't miss activity on a tracked issue. If decisions to not address a
bug aren't tracked in bugzilla, then they can't really do that. I'm
sure Ehsan would have had a conversation if he'd actually remembered
that he had a concern - bugmail would have reminded him :)

You could argue that developers should suck it up and just monitor the
tracking lists manually (your reminders in the weekly meetings are
useful, and I try to do this as much as I can), but I think that
realistically that doesn't scale very well and therefore isn't going
to happen reliably.

I think you're right that this is not a _major_ issue - most of the
time the attention on the tracking list is sufficient to make sure we
don't lose track of something. But on the flip side of that, I'm not
sure I see much downside to explicitly tracking those "won't address"
situations with status:wontfix - it will help Ehsan's use case at
least some of the time (even if in some cases it's "too late"). If the
concern is being able distinguish "certainly will not fix" from
"couldn't fix in time but might want to look at for future respin
possibility", we could add another status or something.

Gavin

> Just so people understand where I'm coming from, it really feels like we're overrotating on something that isn't a major problem and can generally be resolved through conversations when there's concern. Tools and process don't take the place of a conversation saying "this bug really scares me".
>
> -Alex
>
>
> Kyle Huey <m...@kylehuey.com> wrote:On Tue, Jul 10, 2012 at 7:52 AM, akeybl <ake...@mozilla.com> wrote:
>
>> Assigning extra urgency when we build our final beta (or after) is too
>> late in the process. It'd really help if engineering could email us or
>> comment in the bug if they find an issue more critical than the attention
>> it is getting.
>>
>> -Alex
>>
>
> I think the point is that there's no way to tell how critical the issue is
> perceived as from the bug, beyond the binary tracking+/-.
>
> - Kyle

[Alex Keybl]

> It would allow developers to avoid needing to monitor the tracking
> lists closely, and instead rely on their bugmail to make sure they
> don't miss activity on a tracked issue. If decisions to not address a
> bug aren't tracked in bugzilla, then they can't really do that. I'm
> sure Ehsan would have had a conversation if he'd actually remembered
> that he had a concern - bugmail would have reminded him :)

I believe it's a matter of being proactive coming up to the release as opposed to reactive when we mark something as wontfix at the end of the release. Bugmail from comments should do the trick for bringing bugs onto everybody's radar, since we do ping on bugs regularly (for instance https://bugzilla.mozilla.org/show_bug.cgi?id=749010#c58). If there's a way that we can help developers be proactive earlier in the process, I'd be up for making that process change.

-Alex

[Gavin Sharp]

On Tue, Jul 10, 2012 at 8:33 AM, Alex Keybl <ake...@mozilla.com> wrote:
> I believe it's a matter of being proactive coming up to the release as opposed to reactive when we mark something as wontfix at the end of the release.

Fair enough - but being reactive at the end can beat forgetting entirely.

> Bugmail from comments should do the trick for bringing bugs onto
> everybody's radar, since we do ping on bugs regularly (for instance
> https://bugzilla.mozilla.org/show_bug.cgi?id=749010#c58).

Also a fair point, but note that comments don't quite get the same
bugmail-scanning attention that seeing "wontfix" does.

You're right that we're over-rotating on this - sorry for my part in
that. Just trying to represent the overburdened/lazy engineer
perspective :)

Gavin

[Ehsan Akhgari]

On 12-07-10 11:50 AM, Gavin Sharp wrote:
>> Bugmail from comments should do the trick for bringing bugs onto
>> everybody's radar, since we do ping on bugs regularly (for instance
>> https://bugzilla.mozilla.org/show_bug.cgi?id=749010#c58).
>
> Also a fair point, but note that comments don't quite get the same
> bugmail-scanning attention that seeing "wontfix" does.

Yes. This. A thousand times! :-)

After reading this thread I have come to the conclusion that the
tracking flags are used somewhat differently by release drivers and
developers. As a developer, if somebody changes a tracking or status
flag on a bug and I get a bugmail for it, my mind immediately begins to
think "watch this. does this make sense? pay some attention here."
However if I see a bugmail bringing up some nits in a patch, I won't go
back and look at every other bugmail relating to the same bug to see if
it's a tracking+ bug or not.

Also, note that the reason that I brought up this thread was not at all
about bug 749010. I just wanted to see if we can change something in a
way that potential things like that do not fall into the cracks as far
as developers are concerned. In the specific case of that bug, this
issue may not seem too important, but there are tracking+ bugs which
have very severe downsides and a fix would be quite within reach in a
given cycle, and if bug 749010 was one of them, we would be having a
totally different conversation now ("what caused us to ship something
which was known to be broken in this way?"). As a developer, if I see
such as a bug, I make sure it's tracking+, and once it is, I currently
lay back and assume that I'm going to get notified in a bugmail (for
example by seeing that the tracking flag was minused, the status was
changed, etc.) But this thread makes me rethink that expectation.

Cheers,
Ehsan

[Lukas Blakk]

On Jul 11, 2012, at 1:28 AM, Ehsan Akhgari wrote:

> As a developer, if I see such as a bug, I make sure it's tracking+, and once it is, I currently lay back and assume that I'm going to get notified in a bugmail (for example by seeing that the tracking flag was minused, the status was changed, etc.) But this thread makes me rethink that expectation.

Not sure where the 'lay back' part comes in. If something is plussed for tracking that means it should take priority over other work until it's a) landed & uplifted to the appropriate level b) disabled, if that is the goal of the bug or c) untracked/wontfixed. I don't see anything from this conversation that suggested a tracked bug should be treated otherwise by developers.

Alex and I go through our tracked bugs at least once a week, but usually more often, and prune the list as bug statuses change but we tend to leave bugs that are being actively worked on alone. We send out a few nagging emails in a cycle that round up what's on our radar and make it known to the assignees and their managers.

In the last week before release there might be a few bugs that we decide wouldn't block a release but we should still be able to leave them as tracked in the hopes that they might land in time and for any developer who has had a bug in that situation, you've probably noticed that we contact you through many channels (irc, email) to evaluate whether the bug is something we could ship without if it's not going to be ready in time. I don't see anywhere here where the developer is working on something in vain or without awareness of what's happening in the release cycle.

My concern is that if we wontfix a bug too early in the cycle that we are sending the wrong message and lowering our expectations for what can be accomplished in a 6 week cycle. I'd rather see bugs that get tracking have the longest possible time to get worked into the release, even when that means sometimes taking a few fixes in the last beta.

Cheers,
Lukas

[Dao]

On 11.07.2012 15:04, Lukas Blakk wrote:
>
> On Jul 11, 2012, at 1:28 AM, Ehsan Akhgari wrote:
>
>> As a developer, if I see such as a bug, I make sure it's tracking+, and once it is, I currently lay back and assume that I'm going to get notified in a bugmail (for example by seeing that the tracking flag was minused, the status was changed, etc.) But this thread makes me rethink that expectation.
>
> Not sure where the 'lay back' part comes in. If something is plussed for tracking that means it should take priority over other work until it's a) landed & uplifted to the appropriate level b) disabled, if that is the goal of the bug or c) untracked/wontfixed. I don't see anything from this conversation that suggested a tracked bug should be treated otherwise by developers.

The 'lay back' part comes in with the fact that the person requesting
tracking very often isn't one of the developers qualified or responsible
for fixing the bug. So somebody thinks we shouldn't ship Firefox with a
given bug, release drivers agree(?) and set the tracking flag, and then
effectively nothing happens.

[Ehsan Akhgari]

Precisely. I am not always able to fix a bug myself, but I may have
ideas on why we should fix it for the release, so if I get some kind of
an indication that the release is going to ship with that bug not fixed
(let's say the status field being set to an imaginary
"probably-wontfix") then I can take some action on the bug (which again
may not be fixing it, but it could be pinging the responsible people,
trying to think of work-arounds, etc.)

The current way that we track bugs makes this very hard for developers
because they don't triage the tracking list the same way that the
release drivers do.

Ehsan

[Alex Keybl]

> So somebody thinks we shouldn't ship Firefox with a given bug, release drivers agree(?) and set the tracking flag, and then effectively nothing happens.

Can you clarify what you mean by "nothing happens"? A bug going unfixed in a release is not the same as nothing happening. It means that the investigation was not in a position to deliver a fix (because of risk, lack of reproducibility, etc.) at the time of release.

Tracking does not ensure fixing. It ensures investigation.

-Alex

[Alex Keybl]

> Also, note that the reason that I brought up this thread was not at all about bug 749010. I just wanted to see if we can change something in a way that potential things like that do not fall into the cracks as far as developers are concerned.

I'd love to hear suggestions for ways to get your attention earlier in the cycle.

-Alex

On Jul 10, 2012, at 10:28 PM, Ehsan Akhgari wrote:

> On 12-07-10 11:50 AM, Gavin Sharp wrote:
>>> Bugmail from comments should do the trick for bringing bugs onto
>>> everybody's radar, since we do ping on bugs regularly (for instance
>>> https://bugzilla.mozilla.org/show_bug.cgi?id=749010#c58).
>>
>> Also a fair point, but note that comments don't quite get the same
>> bugmail-scanning attention that seeing "wontfix" does.
>
> Yes. This. A thousand times! :-)
>
> After reading this thread I have come to the conclusion that the tracking flags are used somewhat differently by release drivers and developers. As a developer, if somebody changes a tracking or status flag on a bug and I get a bugmail for it, my mind immediately begins to think "watch this. does this make sense? pay some attention here." However if I see a bugmail bringing up some nits in a patch, I won't go back and look at every other bugmail relating to the same bug to see if it's a tracking+ bug or not.
>

> Also, note that the reason that I brought up this thread was not at all about bug 749010. I just wanted to see if we can change something in a way that potential things like that do not fall into the cracks as far as developers are concerned. In the specific case of that bug, this issue may not seem too important, but there are tracking+ bugs which have very severe downsides and a fix would be quite within reach in a given cycle, and if bug 749010 was one of them, we would be having a totally different conversation now ("what caused us to ship something which was known to be broken in this way?"). As a developer, if I see such as a bug, I make sure it's tracking+, and once it is, I currently lay back and assume that I'm going to get notified in a bugmail (for example by seeing that the tracking flag was minused, the status was changed, etc.) But this thread makes me rethink that expectation.
>
> Cheers,
> Ehsan

[L. David Baron]

On Wednesday 2012-07-11 09:04 -0400, Lukas Blakk wrote:
> Not sure where the 'lay back' part comes in. If something is
> plussed for tracking that means it should take priority over other
> work until it's a) landed & uplifted to the appropriate level b)
> disabled, if that is the goal of the bug or c)
> untracked/wontfixed. I don't see anything from this conversation
> that suggested a tracked bug should be treated otherwise by
> developers.

I have two problems with this:

First: There are very few signs that individual bugmail, review
requests, etc., are related to a tracking+ bug. I have enough
trouble reading all my bugmail as it is (i.e., I often don't), and I
certainly don't have time to load each bug (especially given that
that usually takes 10-20 seconds given how slow Bugzilla is) and see
if it's tracking+ while I'm doing it.

Second: Everyone wants to think the thing they prioritize is top
priority, but it can't be. We can't have both tracking+ bugs and
security bugs and bugs critical for Fennec Native / k9o all be top
priority. Additionally, we shouldn't have all of our priorities be
reactive; sometimes we should be prioritizing forward-looking work.
That doesn't mean we shouldn't get the tracking+ bugs fixed in time,
though.

-David

--
𝄞 L. David Baron http://dbaron.org/ 𝄂
𝄢 Mozilla http://www.mozilla.org/ 𝄂

[Alex Keybl]

> First: There are very few signs that individual bugmail, review
> requests, etc., are related to a tracking+ bug. I have enough
> trouble reading all my bugmail as it is (i.e., I often don't), and I
> certainly don't have time to load each bug (especially given that
> that usually takes 10-20 seconds given how slow Bugzilla is) and see
> if it's tracking+ while I'm doing it.

Including Byron and David here to see if they have any suggestions of how to filter bugs marked with the custom tracking flag in email, or whether we can surface that value better.

> Second: Everyone wants to think the thing they prioritize is top
> priority, but it can't be. We can't have both tracking+ bugs and
> security bugs and bugs critical for Fennec Native / k9o all be top
> priority. Additionally, we shouldn't have all of our priorities be
> reactive; sometimes we should be prioritizing forward-looking work.
> That doesn't mean we shouldn't get the tracking+ bugs fixed in time,
> though.

The burden of prioritization should not fall on individual developers, and managers/PMs should be helping out. If this is ever a problem, please do get in contact with us (releas...@mozilla.com). I can say, however, that bugs affecting shipping products (tracked bugs) will almost always get priority over long pole work. If longer pole work is deemed higher priority, we still need to have the conversation so that we can find somebody else to work on the bug.

-Alex

[Johnathan Nightingale]

On Jul 11, 2012, at 1:19 PM, L. David Baron wrote:

> On Wednesday 2012-07-11 09:04 -0400, Lukas Blakk wrote:
>> Not sure where the 'lay back' part comes in. If something is
>> plussed for tracking that means it should take priority over other
>> work until it's a) landed & uplifted to the appropriate level b)
>> disabled, if that is the goal of the bug or c)
>> untracked/wontfixed. I don't see anything from this conversation
>> that suggested a tracked bug should be treated otherwise by
>> developers.
>

> …

> Second: Everyone wants to think the thing they prioritize is top
> priority, but it can't be. We can't have both tracking+ bugs and
> security bugs and bugs critical for Fennec Native / k9o all be top
> priority. Additionally, we shouldn't have all of our priorities be
> reactive; sometimes we should be prioritizing forward-looking work.
> That doesn't mean we shouldn't get the tracking+ bugs fixed in time,
> though.

This thread will fall apart if it becomes a general re-hash of "how to solve the prioritization problem." Tracking+ bugs get regular attention from our release managers and form the space of "things we want to understand about this release." I'd like tracking+ to get developer's attention immediately but, as David says, lots of people think their stuff is important.

Ehsan's initial concern, AIUI, was that we shipped with open tracking+ bugs, which we did, and have in the past, and likely will again. I think his initial concern stemmed from a misunderstanding about the meaning of the flag. The flag means "bugs we are tracking for a particular release" and doesn't guarantee resolution. It's a tool for people like Alex and Lukas to focus their own attention on the proximate, technical risk around bugs. I think they would both welcome any help people can offer in:

a) getting stuff onto their radar sooner
b) helping know which things on the radar they should push hardest, and
c) finding ways to help developers prioritize those bugs.

I'm really excited about those parts of the thread, because we might be able to get to some good changes, I just want to avoid the gravity well.

J

---
Johnathan Nightingale
Sr. Director of Firefox Engineering
@johnath

[Alex Keybl]

> I think the main takeaway from this thread should be the suggestion
> that indicating that the bug is "in trouble" by setting a
> (probably-?)wontfix flag could help ensure those outcomes, in ways
> that nag comments might not.

What I'm hearing here is that we need a way for developers to make sure tracked bug bugmail ends up in their inbox, and there may need to be a behavioral change in attention paid to that mail. We try to combat that with "nag" emails to teams and managers, but I'm also hearing that those may not be enough to get attention.

As mentioned before, granularity of status ("may not fix"?) within the tracking lists will have little effect with the problem I'm seeing here.

-Alex

On Jul 11, 2012, at 11:09 AM, Gavin Sharp wrote:

> On Wed, Jul 11, 2012 at 10:51 AM, Johnathan Nightingale
> <joh...@mozilla.com> wrote:
>> a) getting stuff onto their radar sooner
>> b) helping know which things on the radar they should push hardest, and
>> c) finding ways to help developers prioritize those bugs.
>

> I think the main takeaway from this thread should be the suggestion
> that indicating that the bug is "in trouble" by setting a
> (probably-?)wontfix flag could help ensure those outcomes, in ways
> that nag comments might not.
>
> It might seem silly to play "flag games" for the sake of accommodating
> engineers who can't otherwise prioritize their bugmail properly. Maybe
> it is silly, and there are more effective ways to fix the
> bugmail/buglist attention gap - we should discuss those too! But,
> assuming the cost to implementing this particular proposal is
> relatively low for release-management (I haven't seen this mentioned
> in the thread, so I don't know whether that's a safe assumption), it
> seems like it wouldn't hurt to try.
>
> Gavin

[David Lawrence]

We could certainly add a custom email header such as:

X-Bugzilla-Tracking: tracking-firefox13:? tracking-firefox12:+
status-firefox12:affected

where the values are the field names and their respective set value
concatenated together with a colon. You could then filter on that header
in your client. Would something like that be useful?

dkl

On 07/11/2012 01:46 PM, Alex Keybl wrote:
>> First: There are very few signs that individual bugmail, review
>> requests, etc., are related to a tracking+ bug. I have enough
>> trouble reading all my bugmail as it is (i.e., I often don't), and I
>> certainly don't have time to load each bug (especially given that
>> that usually takes 10-20 seconds given how slow Bugzilla is) and see
>> if it's tracking+ while I'm doing it.
>
> Including Byron and David here to see if they have any suggestions of how to filter bugs marked with the custom tracking flag in email, or whether we can surface that value better.

--
David Lawrence
d...@mozilla.com

[Alex Keybl]

> We're suggesting a way to make sure we see the bugmail. What's wrong with it? Why do you think the costs outweigh the benefits? What do you think the costs *are*?

It has no cost other than having one more flag state to explain, but I have issue with wallpapering over the actual problem here - we need to know when you all find something to be more urgent than the attention it is getting as soon as possible in the cycle.

> This thread has several developers (me, Ehsan, Dao) suggesting that the generated bugmail (if not the granularity itself) would have a positive effect, based on our usage patterns.

This decision (changing status to "may not fix") happens too late in the cycle, and can't be made any earlier due to ongoing investigations. I'd like to find a better solution.

-Alex

On Jul 11, 2012, at 11:42 AM, Gavin Sharp wrote:

> On Wed, Jul 11, 2012 at 11:21 AM, Alex Keybl <ake...@mozilla.com> wrote:
> What I'm hearing here is that we need a way for developers to make sure tracked bug bugmail ends up in their inbox, and there may need to be a behavioral change in attention paid to that mail. We try to combat that with "nag" emails to teams and managers, but I'm also hearing that those may not be enough to get attention.
>

> We're suggesting a way to make sure we see the bugmail. What's wrong with it? Why do you think the costs outweigh the benefits? What do you think the costs *are*?

>
> As mentioned before, granularity of status ("may not fix"?) within the tracking lists will have little effect with the problem I'm seeing here.
>

> What are you basing that assertion on? This thread has several developers (me, Ehsan, Dao) suggesting that the generated bugmail (if not the granularity itself) would have a positive effect, based on our usage patterns. I assume your resistance comes from having identified costs to producing that bugmail, so it might be most useful for you to explain those, rather than just asserting that it won't be a net win and that we need some other kind of solution.
>
> Gavin

[Alex Keybl]

I'd personally love to see this. It'd help get priority bugmail into my inbox. Based upon conversations here, I believe that would be helpful for others as well. Let's hear what they think though.

-Alex

[Steve Wendt]

On 7/11/2012 11:21 AM, Alex Keybl wrote:

> What I'm hearing here is that we need a way for developers to make
> sure tracked bug bugmail ends up in their inbox, and there may need
> to be a behavioral change in attention paid to that mail. We try to
> combat that with "nag" emails to teams and managers, but I'm also
> hearing that those may not be enough to get attention.

Finding a way to automate this using Bugzilla whining sounds like a good
idea. If recipients knew this was always high priority stuff, they
could filter it differently from the rest of the Bugzilla mail.

[Alex Keybl]

We currently send these "nag" emails automatically, and like the fact that it's a per-team roll-up (gives visibility to the whole team about critical issues). Integration into bugzilla can be a longer term goal if others find value, but it would require integration into LDAP to meet the current feature set.

-Alex

[L. David Baron]

On Wednesday 2012-07-11 14:46 -0400, David Lawrence wrote:
> We could certainly add a custom email header such as:
>
> X-Bugzilla-Tracking: tracking-firefox13:? tracking-firefox12:+
> status-firefox12:affected
>
> where the values are the field names and their respective set value
> concatenated together with a colon. You could then filter on that header
> in your client. Would something like that be useful?

It would be somewhat useful to me, in that I'd make it visible
within my mail client (as I do for most of the other X-Bugzilla
headers), and it would show me more information about the state of
the bug as I'm reading bugmail.

(That said, what I really want is a way of reading bugmail that's
not email, and that shows me a lot more about the state of the bug,
but that also doesn't require me to wait for the whole bug to load.)

[Gavin Sharp]

On Wed, Jul 11, 2012 at 11:21 AM, Alex Keybl <ake...@mozilla.com> wrote:

> What I'm hearing here is that we need a way for developers to make sure
> tracked bug bugmail ends up in their inbox, and there may need to be a
> behavioral change in attention paid to that mail. We try to combat that
> with "nag" emails to teams and managers, but I'm also hearing that those
> may not be enough to get attention.
>

[Ehsan Akhgari]

On 12-07-11 2:46 PM, David Lawrence wrote:
> We could certainly add a custom email header such as:
>
> X-Bugzilla-Tracking: tracking-firefox13:? tracking-firefox12:+
> status-firefox12:affected
>
> where the values are the field names and their respective set value
> concatenated together with a colon. You could then filter on that header
> in your client. Would something like that be useful?

No, because not every email client is capable of using custom email
headers. Gmail is a good example.

(And please let's not get into the debate on what email client to use.
Fact of the matter is that some people use clients which can't use
custom headers, and they're not going to change how they use email
because of various reasons.)

Cheers,
Ehsan

[Ehsan Akhgari]

On 12-07-11 12:57 PM, Alex Keybl wrote:
>> So somebody thinks we shouldn't ship Firefox with a given bug, release drivers agree(?) and set the tracking flag, and then effectively nothing happens.
>

> Can you clarify what you mean by "nothing happens"? A bug going unfixed in a release is not the same as nothing happening. It means that the investigation was not in a position to deliver a fix (because of risk, lack of reproducibility, etc.) at the time of release.

It could mean that, but perhaps that's because not enough investigation
has happened (which I think was what happened with the bug which caused
me to start this thread.)

> Tracking does not ensure fixing. It ensures investigation.

It ensures investigation by the release drivers, not necessarily by the
developers throughout the cycle. However, marking something as "may not
get fixed" may raise a red flag in some developer's mind, and gives them
a chance to perform a deeper investigation if they feel that is worth doing.

Ehsan

[Ehsan Akhgari]

On 12-07-11 1:51 PM, Johnathan Nightingale wrote:
> Ehsan's initial concern, AIUI, was that we shipped with open tracking+ bugs, which we did, and have in the past, and likely will again.

Not really. Let me try to state what I think more clearly. I think the
decision made by the release drivers that it's OK to ship Firefox 13
with this tracking+ bug unaddressed (which as far as I can see was *not*
reflected on the bug) was a mistake, one which _could_ have been caught
by the developers CCed on the bug being notified that release drivers do
not think that this bug should be addressed before we ship.

I also think that we ended up getting lucky that not addressing this bug
did not cause problems for a lot of our users.

My concern is that letting developers know that release drivers do not
think that a given tracked bug should be addressed before release makes
it possible that they point out that it's a mistake, try to investigate
more, communicate it with other developers, etc.

> I think his initial concern stemmed from a misunderstanding about the
meaning of the flag. The flag means "bugs we are tracking for a
particular release" and doesn't guarantee resolution. It's a tool for
people like Alex and Lukas to focus their own attention on the
proximate, technical risk around bugs.

I don't believe there is a misunderstanding on the meaning of the flag
on my part. I do believe that there is a misunderstanding on the
release drivers' side that it's OK to ship with a tracking+ bug not
addressed because if it were serious enough, somebody would have brought
it up. I, Gavin, and Dao think that is not necessarily true, because
developers sometimes just miss stuff.

> I think they would both welcome any help people can offer in:
>

> a) getting stuff onto their radar sooner

I'm open to suggestion about this (which should probably happen in
another thread). What I do to help with this is to set bugs that I
think need to be on the radar as tracking as soon as possible.

> b) helping know which things on the radar they should push hardest, and

I agree, but this is not what this thread was about at all (I don't
quite know how this turned into a discussion about prioritization of
work items ;-)

> c) finding ways to help developers prioritize those bugs.

This sort of covers this thread if you read it as making it possible for
developers to complain if a bug doesn't get prioritized high enough.)

Cheers,
Ehsan

[Ehsan Akhgari]

On 12-07-11 2:50 PM, Alex Keybl wrote:
>> We're suggesting a way to make sure we see the bugmail. What's wrong with it? Why do you think the costs outweigh the benefits? What do you think the costs *are*?
>

> It has no cost other than having one more flag state to explain, but I have issue with wallpapering over the actual problem here - we need to know when you all find something to be more urgent than the attention it is getting as soon as possible in the cycle.

Please see my reply to Johnath's email a few minutes ago. I think the
entire discussion about prioritization in this thread is a distraction,
and I would appreciate if people would start new threads to discuss
those matters.

>> This thread has several developers (me, Ehsan, Dao) suggesting that the generated bugmail (if not the granularity itself) would have a positive effect, based on our usage patterns.
>

> This decision (changing status to "may not fix") happens too late in the cycle, and can't be made any earlier due to ongoing investigations. I'd like to find a better solution.

Then how about having an "benign" flag of some sort, which means:
"Release drivers have no reason to believe this bug is very serious that
it absolutely needs to be addressed for a given release, but would
consider taking a fix if it matches the branch safety criteria."

Do you think that status is something that release drivers can get at
sooner in the cycle for a given bug?

Cheers,
Ehsan

[Alex Keybl]

>> Tracking does not ensure fixing. It ensures investigation.
>
> It ensures investigation by the release drivers, not necessarily by the developers throughout the cycle. However, marking something as "may not get fixed" may raise a red flag in some developer's mind, and gives them a chance to perform a deeper investigation if they feel that is worth doing.

Let me put it another way. A tracked/unfixed bug on mozilla-beta is already on the "may not get fixed" list - it's less than 6 weeks until the planned release date, and less than 5 weeks till our final beta build. That's why we send "nag" emails to teams and communicate directly with developers - these bugs already carry significant urgency, and each deserves attention (by multiple developers) all along the way.

My concern is that we may not always have the help of multiple devs on a tracked bug, or get their take on a bug's urgency, even if they are on the CC list. That's the problem I think may be fixable through a minor change to bugmail headers/content and a communication to dev-planning on how to filter on that.

-Alex

On Jul 11, 2012, at 7:12 PM, Ehsan Akhgari wrote:

> On 12-07-11 12:57 PM, Alex Keybl wrote:

>>> So somebody thinks we shouldn't ship Firefox with a given bug, release drivers agree(?) and set the tracking flag, and then effectively nothing happens.
>>

[Alex Keybl]

Let's consider adding that to the top or bottom of the bugmail content then, in support of Gmail users. I believe all modern clients can filter on email content in combination with sender.

-Alex

[Robert Kaiser]

Ehsan Akhgari schrieb:

> On 12-07-11 2:46 PM, David Lawrence wrote:
>> We could certainly add a custom email header such as:
>>
>> X-Bugzilla-Tracking: tracking-firefox13:? tracking-firefox12:+
>> status-firefox12:affected
>>
>> where the values are the field names and their respective set value
>> concatenated together with a colon. You could then filter on that header
>> in your client. Would something like that be useful?
>
> No, because not every email client is capable of using custom email
> headers. Gmail is a good example.

Well, dbaron said it would be useful *to him* at least. Which means it's
useful to *some* people at least and your "no" isn't absolute but only
for a (probably still important) part of the targeted people.
So what it really means is "it's not useful to everyone who needs to
follow this", which means we should investigate at least something in
addition, but doesn't mean that is completely useless.

Robert Kaiser

[David Lawrence]

How about we add it in the header *and* also add similar text to the top
of the bug mail body? That would fill in the gap for Gmail users and
similar. Would the extra body text be considered intrusive to those who
do not care to see that information or need to filter on it?

BTW, I have a bug for tracking this feature (and a preliminary patch for
review):
https://bugzilla.mozilla.org/show_bug.cgi?id=772994

Thanks!
dkl

> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning

--
David Lawrence
d...@mozilla.com

[Nathan Froyd]

On Thu, Jul 12, 2012 at 10:03:25AM -0400, David Lawrence wrote:
> How about we add it in the header *and* also add similar text to the top
> of the bug mail body? That would fill in the gap for Gmail users and
> similar. Would the extra body text be considered intrusive to those who
> do not care to see that information or need to filter on it?

Can we please not add it to the top? The "Do not reply" boilerplate is
useless enough already. Adding it to the bottom should be enough for
filtering based on the content of the email.

-Nathan

[Byron Jones]

David Lawrence wrote:
> How about we add it in the header*and* also add similar text to the top

> of the bug mail body? That would fill in the gap for Gmail users and
> similar. Would the extra body text be considered intrusive to those who
> do not care to see that information or need to filter on it?

i think the extra text _would_ be too intrusive.

we should add it to the headers now, and to the html bugmail in bugzilla
4.2 so we can style it to be less intrusive.

-byron

--
byron - irc:glob - bugzilla.mozilla.org team -

[Ehsan Akhgari]

On 12-07-12 10:03 AM, David Lawrence wrote:
> How about we add it in the header *and* also add similar text to the top

> of the bug mail body? That would fill in the gap for Gmail users and
> similar. Would the extra body text be considered intrusive to those who
> do not care to see that information or need to filter on it?

That's fine by me, but like any other change, there are people who will
object to it, as they have already in this thread. :-)

Ehsan

[Daniel Veditz]

On 7/11/12 7:28 PM, Ehsan Akhgari wrote:
> Then how about having an "benign" flag of some sort, which means:
> "Release drivers have no reason to believe this bug is very serious
> that it absolutely needs to be addressed for a given release, but
> would consider taking a fix if it matches the branch safety criteria."

That category is "not tracking". By default Aurora and Beta are to
"bake" the things we put on mozilla-central. We hope not to take any
fixes whatsoever -- but of course that's unrealistic. If approval is
requested then bug falls in the "would consider taking a fix if it
matches the branch criteria" bucket.

Maybe we want to use the severity "blocker" for bugs that you think
actually block something.

(I'm unclear on whether a tracking "minus" simply means "we've seen
your nomination and we're not tracking it" or a more definite "we
will not take this bug on this branch")

-Dan Veditz

[Alex Keybl]

> (I'm unclear on whether a tracking "minus" simply means "we've seen
> your nomination and we're not tracking it" or a more definite "we
> will not take this bug on this branch")

A minus for tracking usually means that we'd still consider a low risk patch if found, but we typically comment to that end in the bug (especially if that isn't the case).

-Alex

[Benjamin Smedberg]

I've been avoiding this thread, but I can't any more.

On 7/9/2012 2:05 PM, Alex Keybl wrote:
> Tracked bugs are bugs that should be investigated and gain clarity prior to release, but are not guaranteed to be fixed on that same timeline. Bugs are deemed hard blockers closer to release based upon data that we have along with engineering intuition, as opposed to data that we do not. There are no hard and fast rules when we do not know the affected population or general user effect.

I think that this is part of the problem. I think that too often it is
not clear to people watching the bug (as opposed to people in the
release-drivers triage meetings) what the status of a particular bug is.

To take a future example instead of a past example: I nominated bug
771202 for tracking (and it's former clone bug 752340) because I believe
that shipping FF15 with this bug would present an unacceptable web
compatibility regression. My expectation with that bug is that we'll either:

* must fix it before release
* must back out compartment-per-global for FF15 (something that really
needs to happen before beta, and may not even be possible)
* skip the FF15 train

So in this sense I think this should be considered a "blocker". But it's
possible that release-drivers disagrees with me. Let's say that release
drivers marks blocking+ right now. It's not clear from the bug which
decision has been made: the more blocker-y "must be fixed" decision, or
the "we'll watch this and take a fix if appropriate" or even "we can't
decide how serious this is, we've asked X for more data".

We don't necessarily need new bug metadata to solve this issue, but I do
think we need much clearer comments as bugs are marked tracking and then
move along the trains indicating what the possible resolutions of the
tracking bug are.

In this particular case, bholley and I are pretty sure we can get a fix
up before we move to beta on Tuesday (or is it Monday now?), but I think
this is the kind of ambiguity which happens a lot and people make
incorrect assumptions about what release-drivers are actually thinking.

--BDS

[Asa Dotzler]

Benjamin, my expectation is that if it was plussed that would be an
affirmation of the well reasoned request you presented. If it was
minused then you'd probably follow up.

If a requester says "requesting tracking - we simply cannot ship with
this problem because a, b, and c, and our only viable solutions are 1,
2, or 3" and the response from the release team was "yep, plussing" then
presumably the requester would know why it was plussed and the release
team would follow up with the requester about the various options.

It gets more difficult if someone simply requests tracking without
expressing why or what his expectations are. There are plenty of bugs
that get requests like that because they're much more speculative, but
your particular case shouldn't be a hard given the certainty around the
consequences and options. The release team not following up directly
with you when you've made a request like this would definitely be a
failure case but one I think is unlikely in practice.

- A

[Alex Keybl]

> We don't necessarily need new bug metadata to solve this issue, but I do think we need much clearer comments as bugs are marked tracking and then move along the trains indicating what the possible resolutions of the tracking bug are.

I guess this is part of what I'm not understanding. We (Release Management) do try to be very explicit with our comments in bugs, especially how long there is until release, how concerned we are, and possible next steps. People seem to be very passionate about this subject (as is evidence by the number of emails on this thread), but we haven't seen any major fallout from the current tracking process.

As mentioned before, I personally believe what occurred with bug 749010 is more about concerned developers needing to communicate that concern, as opposed to properly setting the bug's status. Current status was "bug unfixed, tracked for this release, X days away from that release", and that was communicated in email/comments. Tracking a bug isn't just "set it and forget it", although we try to bear as much of the follow-up as possible.

> In this particular case, bholley and I are pretty sure we can get a fix up before we move to beta on Tuesday (or is it Monday now?), but I think this is the kind of ambiguity which happens a lot and people make incorrect assumptions about what release-drivers are actually thinking.

Merge days are Monday now, and you can view even more release dates at the Release Management calendar [1].

-Alex

[1] https://mail.mozilla.com/home/ake...@mozilla.com/Release%20Management.html

[Johnathan Nightingale]

On Jul 12, 2012, at 7:58 PM, Alex Keybl wrote:

> As mentioned before, I personally believe what occurred with bug 749010 is more about concerned developers needing to communicate that concern, as opposed to properly setting the bug's status. Current status was "bug unfixed, tracked for this release, X days away from that release", and that was communicated in email/comments. Tracking a bug isn't just "set it and forget it", although we try to bear as much of the follow-up as possible.

I think this is pretty key. Release management does not own bugs. Release management is trying to catch things that fall through the cracks and stay on top of the entire space of a release, but engineers need to own bugs and fixes, and to be engaged and passionate advocates about the stuff that's important. Tools and changes that help make that easier are righteous, but occasionally this thread sounds like people expect release management to do work I actually expect engineers to do - prioritize workload, develop a creative approaches and alternatives, shepherd patches through resolution. tracking+ is a tool we use to gather up all the bugs about a release that need visibility, but it's not a replacement for engineering ownership.

[Byron Jones]

On Jul 11, 2012, at 11:46 AM, David Lawrence wrote:
>> We could certainly add a custom email header such as:
>>
>> X-Bugzilla-Tracking: tracking-firefox13:? tracking-firefox12:+
>> status-firefox12:affected

after some quick coding from david, this change is now live on
bugzilla.mozilla.org.

sample header:
X-Bugzilla-Tracking: tracking-firefox16:+ tracking-firefox17:+


thanks dkl!

[Alex Keybl]

This is awesome! I'll be updating my filters today and will post how I've made use of the headers in a separate dev-planning thread.

-Alex

[Nicholas Nethercote]

On Thu, Jul 19, 2012 at 10:28 AM, Alex Keybl <ake...@mozilla.com> wrote:
> This is awesome! I'll be updating my filters today and will post how I've made use of the headers in a separate dev-planning thread.

Here's how I filter Bugzilla mail in Gmail:
https://blog.mozilla.org/nnethercote/2010/09/16/using-gmail-filters-to-identify-important-bugzilla-mail/

Nick