Fwd: Outstanding Contention Areas for Bugzilla Standardization for QA

[Jason Smith]

Hi Everyone,

Upon thinking about the discussions we had over Bugzilla standardization
for QA, I think we did a good job to figure out how to adopt multiple QA
processes together. However, I do think some of the merits and critiques
of both processes still needs more discussion. Here's my contention
points on my side I still have after thinking about this for a few days:

* The discussion kept harping on the fact that there needs to be
distinction between internal QA involvement vs. out to community.
However, after thinking over this for a few days, I do not think
this distinction is necessary, nor do I think it appropriate to
separate a distinction between QA involvement vs. community
involvement. Separating the distinction creates a barrier between
community involvement and QA, which I think hurts us, not benefits
us (i.e. us vs. them mentality). If a core QA is intended to be
needed, you can easily find this through checking queries daily
under what each Fx release deems as important against common QA
involvement keywords. If an urgent assignee is needed, a needinfo
request against a sheriff can easily raise the importance to find an
assignee quickly. In that argument, I think QA Bugzilla
Standardization should make no distinction between community work
vs. core QA work.
* The discussion made an argument that "qawanted's original intention
was a global definition, so let's stick to it" as being a case for
sticking the keyword on everything. However, the original reason why
B2G QA moved away from this case was that we generally felt like the
qawanted keyword was being stuck on everything with 5 purposes stuck
together. Working now with a process that has pushed more for "state
the purpose," the process benefited from the fact that "purpose" for
involvement was clear. QA Wanted then became the case for when
common purpose requests that aren't covered, so state the fallback
instead. I would additionally argue as well that you should pay
attention generally to any keyword that could warrant QA
involvement, as that shows that QA is paying attention to removing
barriers that allow the bug to be resolved quickly (i.e. should keep
a watch out on regressionwindow-wanted, steps-wanted,
testcase-wanted even if qawanted isn't present). I therefore argue
that we should move away from setting qawanted on every single bug
possible for QA involvement and move to a purpose-driven model. In
that case that where we can't complete the QA request, I'd argue
that the purpose of the QA overseeing the bug is to take alternative
means to figure this out - including getting help from the reporter,
working with the developer to determine a resolution, etc. In some
cases, exhaustion could warrant the keywords being removed and
closing the bug after a time limit is hit to let the reporter take
action on the bug.
* Though not really talked about in discussions that much, I've
noticed the different processes have completely varied processes on
how verifications are handled. On B2G QA, we decided to track
verifications entirely separate from other blocking requests, as
they generally are a backlog we continuously attack as time allows.
We do not view verifications as high priority generally against
other QA requests. I'd actually argue the verifyme keyword is
generally enough to indicate verification would be nice to do. If
the bug has a higher importance, you can easily figure this out with
using the Fx global triage mechanisms with RESOLVED bugs. If a bug
particularly is important to call out for some special case needing
more urgent involvement, I'd actually ping the sheriff directly. I
therefore argue that there needs to be a complete separation of
distinction of priority tracking between bug verifications and other
QA requests.
* Another thing not talked about in the discussion, another general
distinction I see between the different QA processes is a
prioritization model. B2G QA's process model argues for tracking
priorities reflective of the global flags used by the release triage
to determine importance of the bug in an automated manner in
conjunction with other keywords as needed. Desktop QA's process
argues for an explicit qa+ process for tracking high priority
verifications. B2G QA moved originally to a model that derived
priorities from global tracking processes because the prioritization
is automated by global processes, allowing us to integrate better
with the global implications of a project and it's resulting
priorities. I'd therefore argue that priorities for QA involvement
should generally derive from global implications of a project,
rather than individual QA determination of priorities on per bug
basis, mainly on the case for efficiency. If there's a special case,
we can handle those on a case by case basis.

--
Sincerely,
Jason Smith

Desktop QA Engineer
Mozilla Corporation
https://quality.mozilla.com

[Clint Talbert]

CC'ing glob and dkl since they are working on user-role specific work
flows for bugzilla, and this is a gold mine of information w.r.t. how QA
has been thinking of their process as well as (potentially through this
discussion) how QA wants to work going forward.

I love many of these ideas.

On 4/15/2013 6:05 AM, Jason Smith wrote:
>
> Hi Everyone,
>
> Upon thinking about the discussions we had over Bugzilla
> standardization for QA, I think we did a good job to figure out how to
> adopt multiple QA processes together. However, I do think some of the
> merits and critiques of both processes still needs more discussion.
> Here's my contention points on my side I still have after thinking
> about this for a few days:
>
> * The discussion kept harping on the fact that there needs to be
> distinction between internal QA involvement vs. out to community.
> However, after thinking over this for a few days, I do not think
> this distinction is necessary, nor do I think it appropriate to
> separate a distinction between QA involvement vs. community
> involvement. Separating the distinction creates a barrier between
> community involvement and QA, which I think hurts us, not benefits
> us (i.e. us vs. them mentality).

Yes! I think this is crucial in order to truly build a meaningful,
independent community around QA. They cannot feel that they are second
than the employed folks working on QA. They should be peers as merit allows.

> If a core QA is intended to be
> needed, you can easily find this through checking queries daily
> under what each Fx release deems as important against common QA
> involvement keywords. If an urgent assignee is needed, a needinfo
> request against a sheriff can easily raise the importance to find an
> assignee quickly.

Sheriff? Is this a QA sheriff? Or the tree sheriffs? I'm not sure that
the Tree Sheriffs are going to be able to help you much here (though
they would certainly try).

>
> * Another thing not talked about in the discussion, another general
> distinction I see between the different QA processes is a
> prioritization model. B2G QA's process model argues for tracking
> priorities reflective of the global flags used by the release triage
> to determine importance of the bug in an automated manner in
> conjunction with other keywords as needed. Desktop QA's process
> argues for an explicit qa+ process for tracking high priority
> verifications. B2G QA moved originally to a model that derived
> priorities from global tracking processes because the prioritization
> is automated by global processes, allowing us to integrate better
> with the global implications of a project and it's resulting
> priorities. I'd therefore argue that priorities for QA involvement
> should generally derive from global implications of a project,
> rather than individual QA determination of priorities on per bug
> basis, mainly on the case for efficiency. If there's a special case,
> we can handle those on a case by case basis.

I think that your last point is a very important qualification. As QA,
you have to keep development honest. And that means you need to set your
priorities, and not get your priorities dictated to you. Most of the
time, working on the same priorities as development is the right thing
to do, but sometimes, it's not. And you need to be very empowered to
make that case and stand up for that case.

That's my two cents from the peanut gallery. Great email.
Cheers,
Clint

[Jason Smith]

One comment inline for clarification.

Sincerely,
Jason Smith

Desktop QA Engineer
Mozilla Corporation
https://quality.mozilla.com

In B2G QA, we have the concept of a QA sheriff that handles immediate
inquiries for QA support. For example, I might get a needinfo request at
me in order to find an assignee to check for a MMS bug verification.