Is firefox-backlog+ still useful?

[Matthew N.]

I just had a realization that now that we're not using a single backlog for all desktop work (we have various smaller project teams), it's not clear to me what the purpose of the firefox-backlog flag is. There are 1629 bugs set to '+'[1] which seems like a lot given that this flag was intended as a way to mark bugs that we would likely work on in approximately the next 6 months. There are bugs in there that don't seem like they are actually things we want to do.

I'm curious to hear how/if people are currently using the flag and whether anyone is actually pulling bugs from there outside of bugs identified for a specific project. There are searches for good bugs from https://wiki.mozilla.org/Firefox/IterativeDevelopment#Contribute_to_Firefox_Desktop but I don't know if contributors are using them over bugsahoy.

Perhaps the backlog is the place where the upcoming quality teams will pull from as its ideally  identifying important issues in all of our components. A follow-up question would be if anyone is pruning this list?

Cheers,

Matthew N. (:MattN)

[1] https://bugzilla.mozilla.org/buglist.cgi?f1=flagtypes.name&list_id=12437184&o1=equals&query_format=advanced&resolution=---&v1=firefox-backlog%2B&order=bug_id&limit=0

[Christopher Karlof]

I just started using the flag today to build a Desktop Sync backlog.

-chris

_______________________________________________
firefox-dev mailing list
firef...@mozilla.org
https://mail.mozilla.org/listinfo/firefox-dev

[Mark Hammond]

On 1/08/2015 9:34 AM, Christopher Karlof wrote:
> I just started using the flag today to build a Desktop Sync backlog.

I've noticed bugs being changed in this way, and also the same set of
bugs having the "importance" field changed (ie, a number of bugs set to
P1, P2, etc).

I don't understand the use of both of these. Can't the current Sync
backlog be determined based on the importance? What would a P1 bug
without that flag mean relative to a P5 with it?

So I agree with Matt - it became clear to me very soon after the backlog
flag was added that the sheer number of items marked with the flag made
it useless for both filtering and prioritization, and I expect this to
continue to be true for Sync. I don't mind continuing to ignore it
completely, but I'd prefer to just see it die.

Mark

>
> -chris
>
>
> On Fri, Jul 31, 2015 at 2:08 PM, Matthew N.

> <MattN+fi...@mozilla.com <mailto:MattN+fi...@mozilla.com>>

> firef...@mozilla.org <mailto:firef...@mozilla.org>
> https://mail.mozilla.org/listinfo/firefox-dev

[Matthew N.]

I had the same reaction: the priority field overlaps with the backlog flag.

One thing that that the backlog flag has that the priority flag doesn't is a way to nominate a bug for review of the priority via "firefox-backlog?". One could argue that every bug without an assigned priority could fall into this category but then we would want to either regularly triage recently filed bugs to set a priority on them or do a pass on all existing bugs to assign a priority making it easy to see unprioritized bugs. In general I'm not a fan of new BMO fields or whiteboard/keyword metadata that duplicate or overlap with existing metadata.

On a related note, I'm not a fan of fine-grained prioritization for low priority bugs (e.g. like the Rank field is doing). In general, it seems like either a bug is important (represented as P1, maybe P2) or it's not and so thinking about P4 vs. P5 doesn't seem useful as the reality is that neither of them are going to end up in a prioritized backlog. The only way they'll get fixed is if someone is scratching their own itch or the priority changes.

Another "feature" of firefox-backlog+ is that only a limited set of users can set this state. This avoids having to manually police people marking pet bugs as a priority despite the views of the module owner/peers. Maybe the existing BMO groups like canconfirm/editbugs already cover the priority field though (maybe not for bugs by the reporter?).

Matthew

[Sebastian Zartner]

As far as I understand firefox-backlog it is for issues that are lying around for long and need prioritization.[1] So it sounds like it fulfills a different purpose than P1 to P5.

The question is whether this flag is still used as Gavin imagined it to work, i.e. if there's still some backlog management as described in the wiki[2].

I guess it needs to be clarified how the process of prioritizing bugs works and there need to people tasked to do this priorization and keep track of it for all bugs, e.g. the module owners.

My opinion is that the firefox-backlog flag would not be needed if P1, P2, etc. were set consistently and always be considered. E.g. at the moment there are a lot of old bugs having P1.[3]
Also, prioritization needs to be based on many factors like security, number of affected people, severity, votes, other implementations, specifications, etc.

Sebastian

[1] https://mail.mozilla.org/pipermail/firefox-dev/2014-April/001551.html
[2] https://wiki.mozilla.org/Firefox/IterativeDevelopment#Product_Backlog

[3] https://bugzilla.mozilla.org/buglist.cgi?priority=P1&list_id=12440512&resolution=---&chfieldto=2010-12-31&query_format=advanced&chfield=[Bug%20creation]

[Christopher Karlof]

Now that my role has become more of, as my wife says, “one of those people that flips a lot bugzilla flags”, I’m guilty as charged.

I largely adopted it because it was trying to adopt the larger convention, but I agree that filtering searches to P1-Px for a backlog would work equally well of me.

A quick test of that would require me to triage/de-prioritize an additional legacy 100 bugs to get back to the curated backlog I created with firefox-backlog, but that’s the cost of doing business.

I don’t have a lot of context around the original intention of the flag, and am happy to learn any new spells.

-chris

[Justin Dolske]

I also don't think this flag is useful.

The basic problem is that Firefox is a huge, complicated piece of software and the set of people working on it (staff + volunteers) is tiny. The flag was intended as a mechanism to help track relevant bugs that we're not currently working on, but I think we just ended up with (1) varying criteria for what bugs should be marked backlog+ (2) a smaller but still unwieldy set of of bugs and (3) some people simply stopping usage of it because it's effectiveness was unclear.

AFAIK the backlog+ list has never been triaged or prioritized. Nor is it clear to me that would actually be an effective use of time -- it would be spending a lot of time to maintain a wishlist of bugs that will mostly not be worked on. And priorities quickly become stale as the browser market evolves.

So, to make this thread a bit more actionable: I propose that that we go ahead and deprecate this flag. As a first step we can freeze the existing set of bugs with it set, and remove the ability to set/request it on further bugs. Projects areas can track their own project-specific backlogs with priorities, whiteboard tags, metabugs, spreadsheets, etc. If you have a demonstrably useful need for the flag, speak now or forever hold your peace. :)

Justin

[Dave Townsend]

I'm fine with the plan but I'm pretty wary that none of the program managers have said anything, and they're the ones that make use of the flag the most. So I've copied some of them in to make sure they've seen this thread and Justin's proposal to stop using the firefox-backlog flag.

[Mike Connor]

So, the one thing that firefox-backlog+ has that an unrestricted fields don't is an unambiguous signal that this bug you're picking up is something we've said is worth fixing (at some point).  Otherwise it could be something mistagged by the bug filer, etc, and we risk wasting people's time that way instead.

Right now, in theory, the flag state will produce a clear, relatively unambiguous set of bugs that are known to be worth fixing.  I'd want to hear from someone like Mike Hoye who's all about connecting new developers with bugs about how we'd reliably do this without that sort of global filter.

[Sebastian Zartner]

To repeat myself:
Independently of deprecating the backlog flag it needs to be clarified how the process of prioritizing bugs works and there need to people tasked to do this priorization and keep track of it for all bugs.

If there are clear rules across projects for what measures to use for prioritizing bugs, it's easier for contributors to get into that process, especially for bugs affecting several projects.

Sebastian

[Gijs Kruitbosch]

On 12/08/2015 06:29, Mike Connor wrote:

So, the one thing that firefox-backlog+ has that an unrestricted fields don't is an unambiguous signal that this bug you're picking up is something we've said is worth fixing (at some point).  Otherwise it could be something mistagged by the bug filer, etc, and we risk wasting people's time that way instead.

If this is the only reason to keep it around, I bet we could make the Priority, Rank and any other fields that we'd use instead editbugs-only even for bugfilers. It'd save me time and effort when doing triage-type stuff as well.

~ Gijs

[Byron Jones]

Gijs Kruitbosch wrote:

On 12/08/2015 06:29, Mike Connor wrote:

So, the one thing that firefox-backlog+ has that an unrestricted fields don't is an unambiguous signal that this bug you're picking up is something we've said is worth fixing (at some point).  Otherwise it could be something mistagged by the bug filer, etc, and we risk wasting people's time that way instead.

If this is the only reason to keep it around, I bet we could make the Priority, Rank and any other fields that we'd use instead editbugs-only even for bugfilers. It'd save me time and effort when doing triage-type stuff as well.

true.  bugzilla doesn't support it currently, but i know a guy who does perl and could make it happen.


-glob

--
byron jones - :glob - bugzilla.mozilla.org team lead -

[Marco Mucci]

We are going to stop using the flag on the privacy team.

--

Marco Mucci - Engineering Program Manager 

Mozilla Corporation 

Work: 1.416.848.3114 x 860 

Mobile: 647.717.3381 

Email: mmu...@mozilla.com

IRC: MarcoM

Skype: marco.a.mucci

[Mike Connor]

Biggest obstacle there being how to get everyone on board with a global change to BMO. It can be done, but that's something we'll have to discuss in a wider forum.

[Shell Escalante]

Sebastian has a good point about having some clarity into what teams are doing and visibility to contributors (even if it's just teams documenting their method).  I will ask EPM's . There are engineering managers are on this list, who might know people who are working on bugs outside of smaller teams and could ask them how they are managing.

please do not deprecate the flag without a plan. If everyone can live without, then we make a change - but we want a chance for folks to do any mark-up to not lose info if needed.  I don't see any reason why individual teams can't stop using sooner, if it is not adding value to their workload.

several teams use the firefox-backlog flag to show triaged and prioritized bugs versus untriaged bugs - but there might be simpler options.  I also think QA might have it in a few of their queries.  I'll check if they need it and what it does for them.

my teams also set Rank, which we could use instead to know our triaged bugs.  I would love to stop using firefox-backlog - after discussing with teams for OK.  we'd only need time for a bit of clean-up for edge cases. 

I can't use Priority to mark triaged versus untriaged bugs.  Priority has been set on many bugs and not all following the firefox prioritization - and if it was set long ago i'd like to revisit.

Cheers,
Shell

Program Manager | webRTC & Hello

IRC: Shell

[Mike Hoye]

On 2015-08-12 4:36 PM, Shell Escalante wrote:
>
> Sebastian has a good point about having some clarity into what teams
> are doing and visibility to contributors (even if it's just teams
> documenting their method). I will ask EPM's . There are engineering
> managers are on this list, who might know people who are working on
> bugs outside of smaller teams and could ask them how they are managing.

I realize what I'm asking for when I say this, but to the contributor
point: it would be _really_ great for community involvement - I mean,
for a lot of reasons, but that's the one I'm invested in, so - if we
had a reasonably consistent and not-totally-opaque way of expressing
priorities across this organization.


- mhoye

[Benjamin Smedberg]

On 7/31/2015 5:08 PM, Matthew N. wrote:

Perhaps the backlog is the place where the upcoming quality teams will pull from as its ideally  identifying important issues in all of our components. A follow-up question would be if anyone is pruning this list?

I do have a plan for how bugzilla fits in with our broader quality efforts, and this backlog flag is probably not part of it. But I'd like to go into a little more detail about how I'm approaching the problem:

Bug reports are the primary unit of quality assurance in software. Bug reports from our users, but especially pre-release users, are essential to building a quality product. We at Mozilla have lived so long in a world where we don't triage and make a decision about every bug report that we're numb to the pain that it's causing us. We need to make some dramatic changes in how we handle bug triage and the tracking and decision-making process around bugs.

This past quarter we've started small: trying to get the bugs filed each day from Firefox:Untriaged into the right bugzilla component. But the long-term goal is bigger:

• Find or promote a triage and bug handling lead for every bugzilla component. (If no lead can be found, it's probably a sign that the code should become Dead).
  
• Make a crisp decision about every bug.

I've been thinking a lot about what it means to make a decision about every bug, and I don't have final answers, but I do think that we need to end up with the following categories which we can apply to every bug in bugzilla:

• newly filed - next action is component lead
  
• under investigation - clear next step(s) identified
• ready for a product/engineering priority decision - next action is component lead, engineering lead, or product lead
  
• Priority decided
  
• must fix by release N - typically tracked by release drivers or program managers
  
• work backlog for an engineering team - tracked by component lead or other team lead
  
• a valid bug, but not something an engineering team will even prioritize - not tracked
  
• bug currently being fixed (owner identified)
• bug partially fixed (not all QA is done, or not upstreamed/verified on all the necessary branches) - next-step engineering or QA owner identified
  
• bug completely fixed, done - woot!
  
• otherwise closed: wontfix/incomplete/invalid etc
  

We're not very far away from this model right now: there's a lot of fuzz at the beginning of the process and there's a lot of fuzz around how we express bug decision and priorities across our various teams. But mostly the problem is that we don't have enough people triaging bugs, following up on action items, and then make priority decisions.

We're working on that in several ways:

• this week I'm posting a job posting for a bugmaster, which will be a combination program management/QA role to help keep track of components and their owners, create reports, and generally the bugzilla zen garden.
• Bug triage will be a core focus of QA activity later in this year. We are working on a significant QA contract to help with bug triage and bugzilla maintenance, and are continuing efforts to expand community involvement in bug triage and related tasks such as creating testcases, finding regression ranges, etc.
  
• A key part of "great or dead" will be triaging the entire backlog of bugs in related bugzilla components as we get to them. So for instance as we do a great-or-dead review of the Firefox bookmarks system, we will review all of the open bugs in "Firefox:Bookmarks & History" and "Toolkit:Places" so that all the bugs fit into the buckets above at the end of the project.
  

Relative to the specific firefox-backlog bug flag, my experience has been that we don't really use it. We were supposed to pick bugs for the priority backlog from the larger list of firefox-backlog bugs. But in practice, tech leads have in their head the list of bugs they need to accomplish for particular projects, and so the backlog is mostly not looked at once things have been added to it. I don't have a stake in whether we keep it or throw it out for now, but eventually when we have a bugmaster who can spend their full attention figuring out the best way to organize bugzilla flags across multiple teams, I'm hoping we can come up with something more organized and available across multiple teams.

--BDS

[Florin Mezei]

Hello everyone,

 

Speaking for the Desktop Manual QA team at Softvision, we have this field in several of our queries, and it helps us track Sprint work that needs our attention (https://goo.gl/kRtLCx).

 

Here’s an example of query with and without the field… this contains the bugs that we need to monitor in the current 43.1 sprint:

-  WITH the firefox-backlog+ flag – 43 bugs - https://bugzilla.mozilla.org/buglist.cgi?f1=OP&v6=firefox-backlog%2B&o2=substring&o6=substring&f4=cf_qa_whiteboard&query_format=advanced&j1=OR&f3=status_whiteboard&f2=cf_fx_iteration&f5=CP&f6=flagtypes.name&v2=43.1&list_id=12470143

- WITHOUT the firefox-backlog+ flag – 80 bugs - https://bugzilla.mozilla.org/buglist.cgi?f1=OP&o6=substring&o2=substring&f4=cf_qa_whiteboard&query_format=advanced&j1=OR&f3=status_whiteboard&f2=cf_fx_iteration&f5=CP&f6=flagtypes.name&v2=43.1

 

Updating the queries is easy, but if you want to remove the flag it would help if someone could clarify what those extra bugs in the second query are, and how we can filter them out.

 

Regards,

Florin.

 

Florin Mezei | QC Team Lead

SOFTVISION | 57 Republicii Street, 400489 Cluj-Napoca, Romania
Email: fme...@softvision.ro | Web: www.softvision.ro

The content of this communication is classified as SOFTVISION Confidential and Proprietary Information.

[Mark Hammond]

On 13/08/2015 6:17 PM, Florin Mezei wrote:
> Hello everyone,
>
> Speaking for the Desktop Manual QA team at Softvision, we have this
> field in several of our queries, and it helps us track Sprint work that
> needs our attention (https://goo.gl/kRtLCx).
>
> Here’s an example of query with and without the field… this contains the
> bugs that we need to monitor in the current 43.1 sprint:
>
> - WITH the firefox-backlog+ flag – 43 bugs -
> https://bugzilla.mozilla.org/buglist.cgi?f1=OP&v6=firefox-backlog%2B&o2=substring&o6=substring&f4=cf_qa_whiteboard&query_format=advanced&j1=OR&f3=status_whiteboard&f2=cf_fx_iteration&f5=CP&f6=flagtypes.name&v2=43.1&list_id=12470143
>
> - WITHOUT the firefox-backlog+ flag – 80 bugs -
> https://bugzilla.mozilla.org/buglist.cgi?f1=OP&o6=substring&o2=substring&f4=cf_qa_whiteboard&query_format=advanced&j1=OR&f3=status_whiteboard&f2=cf_fx_iteration&f5=CP&f6=flagtypes.name&v2=43.1

At face value, these just seem to be bugs that haven't had that flag
set. I know that's somewhat obvious, but I can't see any reason these
bugs are less "worthy" of QA simply because they lack the flag.

The query is matching iteration="43.1". IIUC, on the Firefox team, if
any bug was marked as being in an iteration, it *must* have had that
flag set - the flag is why it was up for consideration for the
iteration. Marco can correct me if I'm wrong, but any bug with an
iteration set but without that flag simply implies the flag should have
been set.

I don't believe anyone sets that flag with the intent being "this is a
good QA candidate" - which is one of the reasons I'd like to see it die
- the signals it sends to people are often wrong.

I suspect that for your purposes, it might be best to work with
individual program managers (so, eg, you might end up with slightly
different queries per bugzilla component). For example, based on what
Shell indicated, the "Rank" field might be the best indicator for the
stuff she manages, especially if that also allows you to sort by
relative importance. Other teams will do things slightly differently.

Marco does an excellent job of managing the qa-verify? flag, which is
another great indicator and probably extremely useful for you (but not
in your search). But we'd need to arrange for that flag to be
trustworthy (eg, you could needinfo the bug assignee if that flag isn't
set for a bug that otherwise matches your criteria?)

Cheers,

Mark

>
> Updating the queries is easy, but if you want to remove the flag it
> would help if someone could clarify what those extra bugs in the second
> query are, and how we can filter them out.
>
> Regards,
>
> Florin.
>

> *Florin Mezei*| QC Team Lead
>
> *SOFTVISION*| 57 Republicii Street, 400489 Cluj-Napoca, Romania
> *Email: *fme...@softvision.ro <mailto:fme...@softvision.ro> | *Web:
> *www.softvision.ro <http://www.softvision.ro/>

>
> The content of this communication is classified as SOFTVISION
> Confidential and Proprietary Information.
>

> *From:*firefox-dev [mailto:firefox-d...@mozilla.org] *On Behalf
> Of *Shell Escalante
> *Sent:* Wednesday, August 12, 2015 11:37 PM
> *To:* Dave Townsend
> *Cc:* Mooney, Sheila; Jenn Chaulk; Marco Mucci; firefox-dev; Larissa

> Shapiro; Wong, Edwin; Justin Dolske

> *Subject:* Re: Is firefox-backlog+ still useful?

> <https://bugzilla.mozilla.org/buglist.cgi?priority=P1&list_id=12440512&resolution=---&chfieldto=2010-12-31&query_format=advanced&chfield=%5bBug%20creation%5d>

>
> On 3 August 2015 at 04:45, Matthew N.
> <MattN+fi...@mozilla.com

> <mham...@mozilla.com <mailto:mham...@mozilla.com>>

> wrote:
>
> On 1/08/2015 9:34 AM, Christopher Karlof wrote:
>
> I just started using the flag today to build
> a Desktop Sync backlog.
>
>
> I've noticed bugs being changed in this way, and
> also the same set of bugs having the
> "importance" field changed (ie, a number of bugs
> set to P1, P2, etc).
>
> I don't understand the use of both of these.
> Can't the current Sync backlog be determined
> based on the importance? What would a P1 bug
> without that flag mean relative to a P5 with it?
>
> So I agree with Matt - it became clear to me
> very soon after the backlog flag was added that
> the sheer number of items marked with the flag
> made it useless for both filtering and
> prioritization, and I expect this to continue to
> be true for Sync. I don't mind continuing to
> ignore it completely, but I'd prefer to just see
> it die.
>
> Mark
>
>
> -chris
>
>
> On Fri, Jul 31, 2015 at 2:08 PM, Matthew N.
> <MattN+fi...@mozilla.com

> <mailto:MattN%2Bfire...@mozilla.com>
> <mailto:MattN+fi...@mozilla.com
> <mailto:MattN%2Bfire...@mozilla.com>>>

> <mailto:firef...@mozilla.org

[Florin Mezei]

If those bugs are just missing the flag, then indeed we can do without it. I
think a while back the flag was used to separate the work in Marco's team
from other teams that used the same iteration flag (e.g. 43.1), but were of
no interest to us. If that's no longer the case then I have no problem
dropping the backlog flag. Let's see if Marco can clarify this for us.

Regards,
Florin.

-----Original Message-----
From: Mark Hammond [mailto:skippy....@gmail.com]
Sent: Thursday, August 13, 2015 12:16 PM
To: Florin Mezei; 'Shell Escalante'; 'Dave Townsend'
Cc: 'Mooney, Sheila'; 'Jenn Chaulk'; 'Marco Mucci'; 'firefox-dev'; 'Larissa
Shapiro'; 'Wong, Edwin'; 'Justin Dolske'
Subject: Re: Is firefox-backlog+ still useful?

On 13/08/2015 6:17 PM, Florin Mezei wrote:
> Hello everyone,
>
> Speaking for the Desktop Manual QA team at Softvision, we have this
> field in several of our queries, and it helps us track Sprint work
> that needs our attention (https://goo.gl/kRtLCx).
>

> Here's an example of query with and without the field. this contains

> the bugs that we need to monitor in the current 43.1 sprint:
>

> - WITH the firefox-backlog+ flag - 43 bugs -
> https://bugzilla.mozilla.org/buglist.cgi?f1=OP&v6=firefox-backlog%2B&o
> 2=substring&o6=substring&f4=cf_qa_whiteboard&query_format=advanced&j1=
> OR&f3=status_whiteboard&f2=cf_fx_iteration&f5=CP&f6=flagtypes.name&v2=
> 43.1&list_id=12470143
>
> - WITHOUT the firefox-backlog+ flag - 80 bugs -
> https://bugzilla.mozilla.org/buglist.cgi?f1=OP&o6=substring&o2=substri
> ng&f4=cf_qa_whiteboard&query_format=advanced&j1=OR&f3=status_whiteboar

[Marco Mucci]

Hi Florin,

Yes, we previously used the flag to separate work that was important for the whole Desktop team from those that were not.  Now we are in a multi-team situation where whiteboard tags are primarily, but not exclusively, used to identify and separate areas of responsibility. For example, the Privacy Team uses [fxprivacy] and the Partner Search Team uses [fxsearch].  Combined with this we continue to use the Iteration flag to identify which bugs are being worked on in any particular iteration.

Regarding QA, we continue to use 'qe‑verify' flag to mark those bugs which do (+), or do not (-), require verification.  Without the 'firefox-backlog' flag you can still search for bugs tagged for an iteration and if they are marked for verification.  The issue that could arise if someone forgets to set the 'qe‑verify' flag.

Thanks.

Marco

[Florin Mezei]

Thanks Marco! It sounds like we (the Desktop Manual QA team) don’t need the backlog flag anymore. As for the qe-verify flag we constantly monitor fixed bugs in the sprint that don’t have the flag set and make decisions on whether they need verification or not, so we should be good there.

 

Regards,

Florin.

[Shell Escalante]

Thanks Florin, I'll have my teams start being better with the qe-verify flag and we'll stop setting the firefox-backlog flag if that isn't needed for the QA queries.

Thanks BDS, looking forward to someone coming and working on overall bug management.  I think we're getting to a similar end-result through slightly different means.

Cheers,
Shell

Program Manager | webRTC & Hello

IRC: Shell

[Mike Hoye]

On 2015-08-12 1:29 AM, Mike Connor wrote:
> So, the one thing that firefox-backlog+ has that an unrestricted
> fields don't is an unambiguous signal that this bug you're picking up
> is something we've said is worth fixing (at some point). Otherwise it
> could be something mistagged by the bug filer, etc, and we risk
> wasting people's time that way instead.
>
> Right now, in theory, the flag state will produce a clear, relatively
> unambiguous set of bugs that are known to be worth fixing. I'd want
> to hear from someone like Mike Hoye who's all about connecting new
> developers with bugs about how we'd reliably do this without that sort
> of global filter.
>

Up front: nothing I have to say here should be taken as an indictment of
the BMO team, who should have their own bronze statues on pedestals in
the SF office courtyard and possibly a parade.

That said: it is very difficult and frequently impossible to
programmatically make an assertion about the state of a bug, and this
hurts us a lot. This starts at intake [1] with different classes of user
causing bugs to be flagged differently, and rolls downhill from there.

For the sake of argument, let's say that an idealized bug lifecycle is:

1. Intake
2. Investigation,
3. Decision, and if it's a go,
4. Resolution.

The must-have-this, must-have-that thing I've tried to insist on in
"good first" bugs does one job: reducing uncertainty. It's been an
effective tactic, but I think if I'm honest with myself it's still a
half-assed solution; it mostly happens in free-form text boxes. But when
a new contributor looks at those bugs, there's no question about what
that bug needs done to it.

How great would it be if we could all have that at every level?

I think the underlying question here is, how can we mitigate or
eliminate as much uncertainty as possible at every step of a bug's life.

That is: Eliminating uncertainty about bug state and clarifying a bug's
next steps should be _the_ tier-1 design goal here.

Despite being an excellent end goal, I don't think that the
single-flag-state approach will be successful. There's too much work and
state between not-ready and ready, and a major step towards the promised
land of our engineers never even _seeing_ bugs that aren't either
decision-ready or action-ready is being able to know what information
those bugs are lacking.

Some examples off the top of my head:
- When filing bugs the choice of platform defaults to "unspecified"
rather than the platform the user is currently on.
- I can upload a file, but it may or may not be a test case. I don't
know if we can search on "has-test-case".
- If a user reports a crash, we invariably have to go a few rounds back
and forth to ask them to connect us to their crash reports, and that
doesn't always happen.

Small things, but we're losing context and bug-state at every step. That
list is far from exhaustive.

If we had that information and were able to have bugzilla trivially (and
reliably!) search on "has crashreport and has testcase and has
regression-range on my platform", now we have a stack of bugs that are
ready for a module owner to make decisions about.

So instead of starting with flags and agreeing about process and meaning
- and who at Mozilla wouldn't be _super-excited_ to start pushing that
rock up that hill again, I ask you? - I'd like to ask: what does a
Minimum Viable Decision-Ready Bug looks like.

- What does a bug need to be decision-ready?
- How much of that can we capture either automatically, or trivially?

... and then build something that delivers that. I'd be surprised if
that list was a dozen things long.

From my own selfish perspective, being able to search trivially on "bug
on platform X needs Y" (and have a contributor click either "here's X"
or "Bug doesn't really need X") would be a huge non-coding win for
community involvement. We are already driving the number of daily
untriaged bugs down hard with our contributors, and easy access to this
"what does this bug need" state is the obvious next set of steps.

Anyway, there you go. I bet you're glad you asked!

- mhoye




[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1154031

[Gavin Sharp]

> work backlog for an engineering team - tracked by component lead or other team lead

> a valid bug, but not something an engineering team will even prioritize - not tracked

The firefox-backlog flag was intended to represent the "tracking"
you're talking about here, i.e. that distinguishes bugs in these two
buckets.

You are right that it's only useful if the "work backlog" is actually
used, and it's also true that there are other ways to do the tracking
(though it seems to me that they'd likely all boil down to effectively
the same thing). Historically the thing that has been preventing the
backlog from being used has been lack of resourcing/focus, rather than
Bugzilla ergonomics (what flag/annotation is used, how it works etc.).
I'm hopeful that "great or dead" and new resourcing plan will help
address those root problems, because the "how to do it" problem is
easier to solve (and much less important).

Gavin

[Justin Dolske]

On Wed, Aug 12, 2015 at 2:16 PM, Benjamin Smedberg <benj...@smedbergs.us> wrote:

I do have a plan for how bugzilla fits in with our broader quality efforts

\o/
 

Bug reports are the primary unit of quality assurance in software. Bug reports from our users, but especially pre-release users, are essential to building a quality product. We at Mozilla have lived so long in a world where we don't triage and make a decision about every bug report that we're numb to the pain that it's causing us. We need to make some dramatic changes in how we handle bug triage and the tracking and decision-making process around bugs.

A worthy goal, let's just not repeat past mistakes of hoping for big, resource-intensive changes without the realistic people-power and prioritization to actually do it. Headcount for dedicated roles is a fine start, as is the recognition that this is going to be a large change.

 

Relative to the specific firefox-backlog bug flag, my experience has been that we don't really use it. We were supposed to pick bugs for the priority backlog from the larger list of firefox-backlog bugs. But in practice, tech leads have in their head the list of bugs they need to accomplish for particular projects, and so the backlog is mostly not looked at once things have been added to it. I don't have a stake in whether we keep it or throw it out for now, but eventually when we have a bugmaster who can spend their full attention figuring out the best way to organize bugzilla flags across multiple teams, I'm hoping we can come up with something more organized and available across multiple teams.

Thanks. This thread has spawned some good discussion on how we should improve our bug flows. But back to the specific question of the firebox-backlog flag, it feels clearer that:

* It's not filling it's original purpose

* It's doesn't have a clear role in the future of Bugzilla

* At least some groups who we thought might have been using it are not, or misunderstand its meaning

So it still sounds to me like we should get rid of it. I don't think we should conflate this flag with the broader topic of improving everything in Bugzilla -- many of these issues people have noted are long-standing problems (which this flag didn't improve), and keeping it or removing it isn't going to have a material effect in that respect.

Justin

[Matthew N.]

I don't think it would have to be a global change, it could probably be done per-product to avoid having to get universal agreement.

As I mentioned earlier, having a way to nominate a bug for prioritization like firefox-backlog:? would be missing unless we add something like P? alongside P1-P5. Perhaps there is another way to identify these bugs in triage e.g. if every bug gets P1-P5 when triaged and so people making prioritization decisions would search for "---" in the priority field.

-MattN

[Matthew N.]

On Thu, Aug 13, 2015 at 2:15 AM, Mark Hammond <skippy....@gmail.com> wrote:

The query is matching iteration="43.1". IIUC, on the Firefox team, if any bug was marked as being in an iteration, it *must* have had that flag set - the flag is why it was up for consideration for the iteration. Marco can correct me if I'm wrong, but any bug with an iteration set but without that flag simply implies the flag should have been set.

​This duplication/contradiction in BMO fields is one of the reasons why I started the thread: the iteration flag being non-empty currently implies firefox-backlog+ (at least for the Firefox-related BMO products) and so there is redundancy and inconsistency.

I think having fewer fields used more consistently between teams will benefit all contributors and I'm hoping we move in that direction.

-MattN

[Benjamin Smedberg]

On 8/13/2015 6:09 PM, Justin Dolske wrote:
>
>
> A worthy goal, let's just not repeat past mistakes of hoping for big,
> resource-intensive changes without the realistic people-power and
> prioritization to actually do it. Headcount for dedicated roles is a
> fine start, as is the recognition that this is going to be a large change.

Indeed. Working incrementally and solving pieces of the problem instead
of the whole thing at once, while keeping in mind the end-state we
believe in, is going to be some combination of really hard and a fun
challenge.


--BDS

[Gervase Markham]

On 13/08/15 20:04, Mike Hoye wrote:
> Some examples off the top of my head:
> - When filing bugs the choice of platform defaults to "unspecified"
> rather than the platform the user is currently on.

That was a recent and deliberate change; the trouble with setting it to
the user's platform is that it gives the impression that this is the
only platform the problem is seen on. The more general question is the
semantics of the single-select platform field: is it "only this
platform" or "at least this platform"?

> - I can upload a file, but it may or may not be a test case. I don't
> know if we can search on "has-test-case".

There is not an attachment flag for "is_testcase", but if there was (and
defining such would be a simple admin change), you could search for bugs
which had at least one attachment with that flag.

> - If a user reports a crash, we invariably have to go a few rounds back
> and forth to ask them to connect us to their crash reports, and that
> doesn't always happen.

We could do something smart like if they type the word "crash" in the
subject, we pop up extra UI asking for an about:crashes log.

Gerv

[Mike Hoye]

On 2015-08-14 10:32 AM, Gervase Markham wrote:
> On 13/08/15 20:04, Mike Hoye wrote:
>

> [...]

>> - If a user reports a crash, we invariably have to go a few rounds back
>> and forth to ask them to connect us to their crash reports, and that
>> doesn't always happen.
> We could do something smart like if they type the word "crash" in the
> subject, we pop up extra UI asking for an about:crashes log.
>

Right! There are are a lot of relatively easy ways to get this
information, or to make it easier to ask for it.

So far from some casual conversations, I've got:

- Severity of effect
- Testcase / STR
- Related crash reports
- Estimate of affected users
- Addons (or clean-profile/safemode STRs)
- Regression range
- Platform
- Browser version/buildID
- Drivers/HW?


What else do you need to make a go/no-go decision to fix a bug?


- mhoye

[Sebastian Zartner]

On 14 August 2015 at 17:35, Mike Hoye <mh...@mozilla.com> wrote:

On 2015-08-14 10:32 AM, Gervase Markham wrote:

On 13/08/15 20:04, Mike Hoye wrote:

[...]

- If a user reports a crash, we invariably have to go a few rounds back
and forth to ask them to connect us to their crash reports, and that
doesn't always happen.

We could do something smart like if they type the word "crash" in the
subject, we pop up extra UI asking for an about:crashes log.

Right! There are are a lot of relatively easy ways to get this information, or to make it easier to ask for it.

So far from some casual conversations, I've got:

- Severity of effect
- Testcase / STR
- Related crash reports
- Estimate of affected users
- Addons (or clean-profile/safemode STRs)
- Regression range
- Platform
- Browser version/buildID
- Drivers/HW?


What else do you need to make a go/no-go decision to fix a bug?

What about enhancement requests? What is the criteria to prioritize them?

Sebastian

[Chris Peterson]

For those interested in new quality efforts, I recommend checking out Mike Hoye's awesome TriageBot effort. He is crowd-sourcing new bug triage by emailing volunteers one untriaged bug a day. In just three weeks, 120 contributors have reversed the tide from 120–160 new untriaged bugs per day to just five!

https://triagebot.mozilla.org/

[Mike Hoye]

I think it would be a mistake to conflate "knowing what I can work on right now" with "knowing what our organization should build next". While I have strong feelings about that "what should we build next" question - and wow do I ever - I'm 100% sure that shackling both of those questions to the same engine would be a horrible mistake.

For the sake of keeping focus, in this conversation I only care about the (much, much simpler) question of how we can really know what state a bug is in, and how we can leverage that clarity.


- mhoye