Desktop QA: Process for verification, etc.

[Robert Kaiser]

The Firefox Desktop QA team tries to verify all the bugs it possibly can
out of what's fixed in every development cycle.
Unfortunately, we're too few people to cover all bugs that affect the
desktop browser - we already scramble to cover everything in the Firefox
product on Bugzilla, in most cycles, we end up missing up on a lot of
Toolkit and Core changes (which
causes issues like only realizing during beta that we need to test
WebVTT, for example). In addition, I found that apparently different
people in our team are using different queries for that work, which can
be confusing when coordinating across the team.
I also would love us to spend less time on triaging what needs
verification and more instead on the actual QA work.

Based on that, I'm proposing that the desktop QA team will in the future
only look at bugs that either have been fixed in the newly implemented
iteration process of the Firefox team or are flagged to the QA team
explicitly (via verifyme/qawanted, or whatever comes out of the Bugzilla
discussion I also posted about). It should be pretty easy to set up a
single query for this that we can put on some kind of shared dashboard
and so work off a common basis.
If we go that route, we of course need to communicate heavily to
developers that things that need our attention need to be flagged to us.

In addition, I think we should try to ramp up automation for approvals
where possible, so we can focus on bugs that actually require manual
approval.

I'd like to hear what other people in QA or Firefox development are
thinking about this proposal or what alternatives you might see to it.
(Please post discussion items on dev-quality, I'll post to the other
lists as well when we reach a conclusion.)

Robert Kaiser

[Anthony Hughes]

Anything we can do to reduce the triage load reduces the risk of QA missing something which improves the overall product quality. You could argue this contributes to Mozilla's "increase browser usage by 10%" goal as improved product quality likely means people switching away from Firefox less often.

> are flagged to the QA team explicitly (via verifyme/qawanted, or whatever comes out of the Bugzilla discussion I also posted about).

I agree that QA carries too much of the burden of responsibility to ensure nothing important slips through the cracks. I would prefer that we make this a shared responsibility and not offload it completely to developers. If have an intuitive way for developers to put bugs on our radar then it makes our jobs easier and reduces the likelihood that something gets missed. That'd definitely be a big win for overall product quality, but I don't think it would wash QA's hands of the responsibility to triage those bug queries of fixed bugs; it just makes those queries a bit more manageable.

> In addition, I think we should try to ramp up automation for approvals
> where possible, so we can focus on bugs that actually require manual
> approval.

I agree with this too. I think we've been doing this triage process long enough that we can probably vet some criteria to either auto-verify or auto-wontverify certain bugs.

Thanks,

Anthony Hughes
QA Engineer, Desktop Firefox
Mozilla Corporation

> _______________________________________________
> dev-quality mailing list
> dev-q...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-quality
>