The Firefox desktop team is heavily relying on bugs being marked in some way to know if we need to verify them or perform some other action on them. For that, our QA drivers for example triage the bugs fixed in every development cycle and mark them "verifyme" or [qa-]. I've also seen [qa+] float around, and nowadays the whiteboard tags are even sometimes in the normal whiteboard and sometimes in the newly created QA whiteboard field.
For bugs we need to investigate, the "qawanted" keyword is also in use to get our attention.

I wonder if it's really a good idea to have this mixed bag of whiteboards and keywords floating around or if we should consolidate that, for example to using keywords for everything ([qa-] could be replaced by a new "noverify" or similar keyword).
I think the Bugzilla DB is better at queries for keywords than whiteboard tags, as keywords are structured, also it's harder to mistype them, so I personally prefer them over whiteboard tags for cases like this.

What's your opinion on that?
(Please post discussion items on dev-quality, I'll post to the other lists as well when we reach a conclusion.)

Robert Kaiser

Thanks for posting this, Robert.

I've been giving this much thought lately as I've been reflecting on the
Firefox Dev team's new 2-week iteration model. I'd like to see if we can
agree on a system that works for devs, release managers, and individual
QA teams alike. I agree that we need to make things a bit more unified
and clear, less redundant.

The problems I have with existing mechanisms:
  * there are multiple ways to say the same thing
  * there's no good way to differentiate between Desktop and Mobile when
needed
  * they're are hard to understand without prior learning
  * they don't encourage much participation from outside QA
  * they are too ambiguous to allow for much automation/scripting of
common tasks

I would like whatever we come up with to solve all of these problems.

I wonder if we might entertain the idea of just moving everything "QA"
related into the QA Whiteboard. We could design a system of tags that
works well to improve transparency and reduce ambiguity. I know keywords
are designed to be a bit more "fool-proof" so maybe we could make the QA
Whiteboard into a keyword type field instead of something that is more
free-form as it exists today.

That aside, whatever we decide upon will need to be documented and
communicated well, and we'll need to allow for time to educate people on
the best practices. Of course I'll do my best to contribute to this effort.
> _______________________________________________
> dev-quality mailing list
> dev-q...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-quality

--
Anthony Hughes
QA Engineer, Desktop Firefox
Mozilla Corporation

PS

Here's a list of the systems I know we are using today.

  * [qa+] whiteboard tag, typically used to tag something as on our radar
  * [qa-] whiteboard tag, typically used to tag something as off our radar
  * [qa?] whiteboard tag, typically used to tag something as needing
review from us
  * [qa!] whiteboard tag, typically used to tag something we have
finished with
  * we also have verifyme, qawanted, qaurgent, qablocked keywords

Also, this may be out of scope for the discussion here but I'd like to
see if we can come up with a way to solve a couple other problems:

1) Resolving the currently broken link between Target Milestone and
status flags
2) Implementing some automated system for auto-triaging bugs we know
fall into certain QA buckets

Robert Kaiser wrote on 04/16/2014 04:56 PM:

>> 1) Resolving the currently broken link between Target Milestone and
>> status flags
>

> I do not think we can solve this or should even try to do that in a QA
> discussion, as that's something spanning all of our processes at Mozilla.

And the final decision (once we get one) should be put onto bug 975400
(https://bugzilla.mozilla.org/show_bug.cgi?id=975400)

--
Henrik Skupin
Software Engineer in Test
Mozilla Corporation

Anthony Hughes schrieb:

> I wonder if we might entertain the idea of just moving everything "QA"
> related into the QA Whiteboard.

I'd prefer something where we use keywords and/or flags for the main
things, augmented with QA whiteboard for further specification (e.g.
"this is something that can be verified without deep knowledge",
possibly things that automation can hook up to, specifying that it only
affects a certain product, etc.)

I agree that it would be good to find a system that applies well enough
that we can solve all the areas you mentioned.

That said, "hard to understand without prior learning" will surely not
fully go away because we are a complex project, we can help it with
clear and easy documentation about a thought-through system, though.
And "encourage participation from outside QA" can be solved with a
well-documented system and easy-to-find queries as well.

I'd like to get into a state where we have a dashboard that enables us
to see easily what the current status of QA is and what needs work, and
we should also have one-and-done tasks with good explanations for a
number of the things to do there.

Of course that also ties in with the process thread I also started here. :)

KaiRo

Anthony Hughes schrieb:

> 1) Resolving the currently broken link between Target Milestone and
> status flags

I do not think we can solve this or should even try to do that in a QA
discussion, as that's something spanning all of our processes at Mozilla.

KaiRo

What I understand about the current use of keywords and tags is:

1.       Tags ([qa+], [qa-], [qa?], [qa!])

a.       Express the QA status of the bug (qa needed, qa not needed, qa
review needed, qa completed)

b.      They seem to be a fast way to let us know what level of QA work is
needed

c.       Seem particularly useful for the QA team, not as much for
contributors

d.      Are currently used in both Whiteboard and QA Whiteboard, which does
not make sense

2.       Keywords (qawanted, regressionwindow-wanted, steps-wanted,
verifyme.)

a.       Express with more granularity what kind of QA work is needed

b.      Seem particularly useful in maintaining individual queries,
especially for Bug Investigation and Bug Verification

c.       Appear more intuitive for contributors

At the moment, I see two plausible solutions for simplifying the use of
keywords and tags:

1.       Keep using both keywords and tags with the following rules:

a.       Only use the current 4 tags ([qa+], [qa-], [qa?], [qa!]) to express
QA status of bug (qa needed, qa not needed, qa review needed, qa completed)
throughout the iteration process and even outside of it

b.      Use the 4 tags only in the QA Whiteboard field (as tags or keywords)

c.       Keep using the current flow for the tags: [qa?] when QA needs to
review the bug -> [qa+]/[qa-] once need for QA work is clear -> [qa!] once
QA work is done and bug is verified

d.      Use keywords under the following circumstances:

                                                               i.
qawanted/qaurgent/qablocked - do NOT use them anymore since [qa+] expresses
the same thing, and severity/urgency can also be expressed in a comment;
"qawanted" may still be used as a generic keyword to suggest need for QA
help, but this may be replaced by the use of [qa+], comments, and
"need-info"

                                                             ii.
regressionwindow-wanted - use same as we do currently, but do NOT set an
additional flag for [qa+], or qawanted keyword; when regression window is
provided, QA will remove keyword (will allow QA to easily monitor bugs where
regression window is requested)

                                                            iii.
steps-wanted - use same as we do currently, but do NOT set an additional
flag for [qa+], or qawanted keyword; when steps are provided, QA will remove
keyword (will allow QA to easily monitor bugs where steps are requested)

                                                           iv.      verifyme
- use same as we do currently, but do NOT set an additional flag for [qa+],
or qawanted keyword; when bug is fully verified, QA will remove keyword
(will allow QA to easily monitor bugs that need verification)

2.       Use only keywords, and add new keywords that express the same
things as the tags themselves (but in a more intuitive manner). For example:
[qa+]=qa-needed, [qa-]=qa-not-needed, [qa?]=qa-review-needed,
[qa!]=qa-completed. Then the use of a keywords only system would become:

a.       qa-needed=qawanted/qaurgent/qablocked - used whenever qa is needed
for anything (e.g. the developer wants someone to try something, or qa is
needed to monitor a bug throughout the iteration phase); would basically
combine the use of the [qa+] tag and qawanted keyword

b.      regressionwindow-wanted - use same as we do currently; when
regression window is provided, QA will just remove keyword

c.       steps-wanted - use same as we do currently; when steps are
provided, QA will just remove keyword

d.      verifyme - use same as we do currently; when bug is fully verified,
QA will replace the keyword with qa-completed

e.      qa-not-needed - would work same as [qa-] currently

f.        qa-review-needed - would work same as [qa?] currently

To me the second option seems like the simple and efficient one. and would
also enable us to do what Robert was also suggesting in a previous email:

"I'd prefer something where we use keywords and/or flags for the main
things, augmented with QA whiteboard for further specification (e.g. "this

is something that can be verified without deep knowledge",").

 

Feel free to comment on this, and let's find a solution.

 

Regards,

Florin Mezei.

Florin Mezei wrote:
> a.       Only use the current 4 tags ([qa+], [qa-], [qa?], [qa!]) to express
> QA status of bug (qa needed, qa not needed, qa review needed, qa completed)
> throughout the iteration process and even outside of it

IMHO, status of any kind that can only be a fixed number of values is
something that should be in status flags but not in a whiteboard.
Whiteboard should be only for temporary stuff and freetext markers,
possibly for prototyping things that go into proper flags or keywords in
the long run.

Also, we already represent some form of QA status with status fields or
flags being "verified" or not.

> 2.       Use only keywords, and add new keywords that express the same
> things as the tags themselves (but in a more intuitive manner). For example:
> [qa+]=qa-needed, [qa-]=qa-not-needed, [qa?]=qa-review-needed,
> [qa!]=qa-completed.

I think that "qa-(not-)needed" is too generic. Does that mean the
probably in the bug does need to be investigated (i.e. QA is needed to
do something before this can be fixed) or that the fix needs to be
verified (QA needed after fixing)?
That's also one reason why I'm not too happy with the [qa+]/[qa-] system
in general.

If we want to come up with generic bug queries that tell us what work is
on the plate of QA (and what we have done recently) and put those on
dashboards, then we need to differentiate at least between investigation
and verification because it happens on different bug states.

> f.        qa-review-needed - would work same as [qa?] currently

I meant to ask this for a while already: What exactly does this mean anyhow?

> To me the second option seems like the simple and efficient one.

I (obviously) agree on that in principle but think we should have a set
that differentiates between investigation and verification, and we
should make sure all variants make sense.

Another non-keyword way we could do things with is to have a "qa" status
flag on bugs that would get set to different values during the lifetime
of a bug to indicate its QA status - but then, "verified" is already in
the main status flags, and what other statuses do we actually need (i.e.
is absence of a keyword saying enough as well)?

KaiRo

Between Anthony and Florin, it appears that we have the following
attributes:
* qa+, qa-, qa?, qa!
* regressionwindow-wanted
* qawanted
* steps-wanted
* verifyme
* qablocked
* qaurgent

I'd prefer we find a way that streamlines our reporting and makes it
easy for folks not involved with QA to know what is going on. That way
it will be helpful for contributors. I like Florin's more verbose
replacements for example.  However, simply having a proliferation of
these things will be counterproductive too.

Can we agree on what the set of states are that we need to track that
solves Anthony and Robert's needs and then determine what from those
states should be keyword vs qa whiteboard?

Clint

On 4/23/2014 08:19, Florin Mezei wrote:
> What I understand about the current use of keywords and tags is:
>
> 1.       Tags ([qa+], [qa-], [qa?], [qa!])
>

> a.       Express the QA status of the bug (qa needed, qa not needed, qa
> review needed, qa completed)
>

> b.      They seem to be a fast way to let us know what level of QA work is
> needed
>
> c.       Seem particularly useful for the QA team, not as much for
> contributors
>
> d.      Are currently used in both Whiteboard and QA Whiteboard, which does
> not make sense
>
> 2.       Keywords (qawanted, regressionwindow-wanted, steps-wanted,
> verifyme.)
>
> a.       Express with more granularity what kind of QA work is needed
>
> b.      Seem particularly useful in maintaining individual queries,
> especially for Bug Investigation and Bug Verification
>
> c.       Appear more intuitive for contributors
>
> At the moment, I see two plausible solutions for simplifying the use of
> keywords and tags:
>
> 1.       Keep using both keywords and tags with the following rules:
>

> a.       Only use the current 4 tags ([qa+], [qa-], [qa?], [qa!]) to express
> QA status of bug (qa needed, qa not needed, qa review needed, qa completed)
> throughout the iteration process and even outside of it
>

> b.      Use the 4 tags only in the QA Whiteboard field (as tags or keywords)
>
> c.       Keep using the current flow for the tags: [qa?] when QA needs to
> review the bug -> [qa+]/[qa-] once need for QA work is clear -> [qa!] once
> QA work is done and bug is verified
>
> d.      Use keywords under the following circumstances:
>
>                                                                 i.
> qawanted/qaurgent/qablocked - do NOT use them anymore since [qa+] expresses
> the same thing, and severity/urgency can also be expressed in a comment;
> "qawanted" may still be used as a generic keyword to suggest need for QA
> help, but this may be replaced by the use of [qa+], comments, and
> "need-info"
>
>                                                               ii.
> regressionwindow-wanted - use same as we do currently, but do NOT set an
> additional flag for [qa+], or qawanted keyword; when regression window is
> provided, QA will remove keyword (will allow QA to easily monitor bugs where
> regression window is requested)
>
>                                                              iii.
> steps-wanted - use same as we do currently, but do NOT set an additional
> flag for [qa+], or qawanted keyword; when steps are provided, QA will remove
> keyword (will allow QA to easily monitor bugs where steps are requested)
>
>                                                             iv.      verifyme
> - use same as we do currently, but do NOT set an additional flag for [qa+],
> or qawanted keyword; when bug is fully verified, QA will remove keyword
> (will allow QA to easily monitor bugs that need verification)
>

> 2.       Use only keywords, and add new keywords that express the same
> things as the tags themselves (but in a more intuitive manner). For example:
> [qa+]=qa-needed, [qa-]=qa-not-needed, [qa?]=qa-review-needed,

> [qa!]=qa-completed. Then the use of a keywords only system would become:
>
> a.       qa-needed=qawanted/qaurgent/qablocked - used whenever qa is needed
> for anything (e.g. the developer wants someone to try something, or qa is
> needed to monitor a bug throughout the iteration phase); would basically
> combine the use of the [qa+] tag and qawanted keyword
>
> b.      regressionwindow-wanted - use same as we do currently; when
> regression window is provided, QA will just remove keyword
>
> c.       steps-wanted - use same as we do currently; when steps are
> provided, QA will just remove keyword
>
> d.      verifyme - use same as we do currently; when bug is fully verified,
> QA will replace the keyword with qa-completed
>

> e.      qa-not-needed - would work same as [qa-] currently

>
> f.        qa-review-needed - would work same as [qa?] currently
>

> To me the second option seems like the simple and efficient one. and would
> also enable us to do what Robert was also suggesting in a previous email:
> "I'd prefer something where we use keywords and/or flags for the main
> things, augmented with QA whiteboard for further specification (e.g. "this
> is something that can be verified without deep knowledge",").
>
>  
>
> Feel free to comment on this, and let's find a solution.
>
>  
>
> Regards,
>
> Florin Mezei.
>