Proposal for improving bug fix verification rate

[Anthony Hughes]

To those of you who might not know me, my name is Anthony and I co-lead QA release sign-off for Firefox Desktop releases (ashughes on IRC). The QA team has been trying to find a way to better prioritize, visualize, and increase traction of bug fix verifications; especially as it pertains to tracked issues for Firefox releases. One of the biggest efficiency sinks is the back and forth that inevitably happens when we triage the hundreds of fixed bugs each release trying to decide if and how a bug fix should be verified. The following proposal outlines three initiatives we hope will improve this process.

Part 1: VERIFYME Flag
------------------------------
The first part of our proposal is to create a new bugzilla flag called "verifyme". The purpose would be to help us prioritize bug verification work and minimize time lost due to triaging unverifiable fixes and unnecessary verifications. The flag would replace the [qa+-?] whiteboard tags with three possible states:

* verifyme+: fix is verifiable and needs to be verified
* verifyme-: fix is unverifiable or unnecessary to verify
* verifyme?: QA needs further instruction before verification (steps to reproduce, minimized test case, environment variables, etc)

QA would routinely use these flags in our day-to-day triage but I would also encourage their use by engineers. When you mark a bug RESOLVED FIXED and you know the fix needs verification, you could set the verifyme+ flag. Alternatively, you could set the verifyme- flag when you know the FIXED bug is unverifiable or doesn't need verification. Doing so would mitigate the occurrences of QA asking how to verify or whether a bug needs verification (a common occurrence and hurdle).

Part 2: VERIFYME Keyword
-------------------------
The second part of our proposal is for developers, engineers, managers, testers, or anyone else engaged in a bug to get used to using the "verifyme" keyword which already exists in Bugzilla. The purpose would be to indicate priority to QA. In other words, QA would triage the verifyme keyworded bugs first and foremost. We believe this will reduce the occurrence of important fixes being released untested.

Part 3: QA Contact Field
------------------------
The third and final part of our proposal is for QA to use the QA Contact field to indicate assignment. We've already started to use the field to assign qawanted keyworded bugs to individual testers and we'd like to start using it to similarly assign responsibility for fix verification. We believe this will provide greater accessibility and visibility of QA involvement. It also creates a distinctive contributor path in that it allows volunteers to take ownership of verifying fixes.

Thank you for taking your time to consider our proposal. At this point I'll open up the floor to anyone who has feedback, questions, or concerns with this proposal.

Cheers,

Anthony Hughes
Quality Engineer
Mozilla Corporation

[Justin Lebar]

So from a developer's perspective, the only change you're asking for
here is that we use [verifyme-] to indicate when a bug cannot be
manually verified?

Are you asking that we put [verifyme-] on all RESOLVED FIXED bugs, or
only some? If some, which ones?

Can developers use [verifyme+] when they explicitly want verification,
or should we use [verifyme?]?

-Justin

> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning

[Anthony Hughes]

Hi Justin, thanks for the quick response.

> use [verifyme-] to indicate when a bug cannot be manually verified?

Correct.

> Can developers use [verifyme+] when they explicitly want verification

Certainly, in fact this would be encouraged.

> should we use [verifyme?]

Only in cases where you know a bug can/should be manually verified but lacks something needed to complete the verification (steps, testcase, etc).

Does that make sense?

[Justin Lebar]

> Does that make sense?

Yes, but what about

> Are you asking that we put [verifyme-] on all RESOLVED FIXED bugs [that
> cannot be verified], or only some? If some, which ones?

In other words, which bugs are candidates for verification? If the
proposal is that I put verifyme- on all bugs that don't need
verification (98% of my bugs), that would be kind of annoying.

[Jorge Villalobos]

Maybe it's bikeshedding the topic a little, but shouldn't the flag be
named needsverify or something similar?

Because of the proposed name [verifyme?] reads to me like a verification
is being requested and the + - states can be similarly misinterpreted.

Jorge

[Jorge Villalobos]

Maybe it's bikeshedding the topic a little, but shouldn't the flag be
named needsverify or something similar?

Because of the proposed name [verifyme?] reads to me like a verification
is being requested and the + - states can be similarly misinterpreted.

Jorge

On 8/9/12 4:50 PM, Anthony Hughes wrote:

[Anthony Hughes]

> If the proposal is that I put verifyme- on all bugs that don't need verification (98% of my bugs), that would be kind of annoying.

I could see how that would be annoying. That said, it's just as annoying for QA to have to ask on 98% of your bugs if we need to verify it. That said, I'm not trying to propose something developers *must* use, rather that they can use the flag if they find it useful. Think of it as more of a tool you can use to cut down on unnecessary bug traffic. I'd ultimately leave it up to your discretion. QA will continue to triage unflagged bugs, however flagged bugs will be given priority.

----- Original Message -----
From: "Justin Lebar" <justin...@gmail.com>
To: "Anthony Hughes" <ahu...@mozilla.com>
Cc: "dev-planning" <dev-pl...@lists.mozilla.org>
Sent: Thursday, August 9, 2012 4:13:42 PM
Subject: Re: Proposal for improving bug fix verification rate

> Does that make sense?

Yes, but what about

> Are you asking that we put [verifyme-] on all RESOLVED FIXED bugs [that
> cannot be verified], or only some? If some, which ones?

In other words, which bugs are candidates for verification? If the
proposal is that I put verifyme- on all bugs that don't need
verification (98% of my bugs), that would be kind of annoying.

[Nicholas Nethercote]

On Thu, Aug 9, 2012 at 4:13 PM, Justin Lebar <justin...@gmail.com> wrote:
>
> In other words, which bugs are candidates for verification? If the
> proposal is that I put verifyme- on all bugs that don't need
> verification (98% of my bugs), that would be kind of annoying.

I was wondering a similar thing. I genuinely don't understand the
current process for deciding which bugs get verification. Like
jlebar, a tiny fraction of the bugs I work on are verified, and the
verification rarely (IMO) adds value. (I'm not claiming my experience
is universal, however.)

Maybe the distinction is meant to be that bugs reported by users
should be verified, but bugs reported by developers should not? Any
clarification about this would be appreciated!

Nick

[Anthony Hughes]

> [verifyme?] reads to me like a verification is being requested

There was some debate about that in QA and we decided that's really what the verifyme keyword is meant to indicate (also included in my proposal).

> the + - states can be similarly misinterpreted.

I would think the descriptions I provided earlier would be included in the Bugzilla and would hopefully eliminate some of this misinterpretation.

> Maybe it's bikeshedding the topic a little, but shouldn't the flag be named needsverify or something similar?

We are open to suggestions of flag names which make it clearer. However, I don't want to bikeshed on the naming at this point. I guess what I'm looking for is whether people have objections to, or would get use out of have a flag that helps prioritize bug verification work and cuts down on bug traffic; for example, QA asking if we need to verify the bug.

[Anthony Hughes]

Not sure if the adds more confusion or helps alleviate the confusion but let me try to clarify. QA triages every RESOLVED FIXED bug which is marked as status-firefoxN:fixed for a given release while in Beta. Many of these bugs cannot be verified or are unnecessary to verify.

In a given release there could be as many as 300 bugs, half of which we end up verifying, the rest sit there unverified after some back and forth about whether we need to verify a bug and/or how we can test the fix. We rarely end up reopening a bug, but it does happen, and this is the value of the verification process. This is especially important for security fixes.

This is really about trying to work more efficiently, spending less time on bugs we don't need to, and more time on important bugs.

----- Original Message -----
From: "Nicholas Nethercote" <n.neth...@gmail.com>
To: "Justin Lebar" <justin...@gmail.com>
Cc: "Anthony Hughes" <ahu...@mozilla.com>, "dev-planning" <dev-pl...@lists.mozilla.org>
Sent: Thursday, August 9, 2012 4:22:22 PM
Subject: Re: Proposal for improving bug fix verification rate

[Justin Lebar]

> That said, I'm not trying to propose something developers *must* use, rather
> that they can use the flag if they find it useful. Think of it as more of a tool
> you can use to cut down on unnecessary bug traffic. I'd ultimately leave it up
> to your discretion. QA will continue to triage unflagged bugs, however flagged
> bugs will be given priority.

Okay, I appreciate that it's a request. :) Suppose I'm lazy and
don't want to put verifyme- on all bugs which don't need verification.
But I want to help QA out by putting verifyme- on some bugs.

Is there some heuristic I can apply to know which bugs are most likely
to get flagged for verification, so I can nip those particular ones in
the bud?

> ----- Original Message -----
> From: "Justin Lebar" <justin...@gmail.com>
> To: "Anthony Hughes" <ahu...@mozilla.com>

> Cc: "dev-planning" <dev-pl...@lists.mozilla.org>
> Sent: Thursday, August 9, 2012 4:13:42 PM
> Subject: Re: Proposal for improving bug fix verification rate
>

>> Does that make sense?
>
> Yes, but what about
>
>> Are you asking that we put [verifyme-] on all RESOLVED FIXED bugs [that

>> cannot be verified], or only some? If some, which ones?

>
> In other words, which bugs are candidates for verification? If the
> proposal is that I put verifyme- on all bugs that don't need
> verification (98% of my bugs), that would be kind of annoying.
>

> On Thu, Aug 9, 2012 at 6:50 PM, Anthony Hughes <ahu...@mozilla.com> wrote:
>> Hi Justin, thanks for the quick response.
>>
>>> use [verifyme-] to indicate when a bug cannot be manually verified?
>>
>> Correct.
>>
>>> Can developers use [verifyme+] when they explicitly want verification
>>
>> Certainly, in fact this would be encouraged.
>>
>>> should we use [verifyme?]
>>
>> Only in cases where you know a bug can/should be manually verified but lacks something needed to complete the verification (steps, testcase, etc).
>>
>> Does that make sense?
>>

>> ----- Original Message -----
>> From: "Justin Lebar" <justin...@gmail.com>
>> To: "Anthony Hughes" <ahu...@mozilla.com>

>> Cc: "dev-planning" <dev-pl...@lists.mozilla.org>
>> Sent: Thursday, August 9, 2012 3:44:48 PM
>> Subject: Re: Proposal for improving bug fix verification rate
>>

[L. David Baron]

On Thursday 2012-08-09 16:22 -0700, Nicholas Nethercote wrote:
> On Thu, Aug 9, 2012 at 4:13 PM, Justin Lebar <justin...@gmail.com> wrote:
> >

> > In other words, which bugs are candidates for verification? If the
> > proposal is that I put verifyme- on all bugs that don't need
> > verification (98% of my bugs), that would be kind of annoying.
>

> I was wondering a similar thing. I genuinely don't understand the
> current process for deciding which bugs get verification. Like
> jlebar, a tiny fraction of the bugs I work on are verified, and the
> verification rarely (IMO) adds value. (I'm not claiming my experience
> is universal, however.)

I think deciding this depends on knowing what the intended purpose
of verification is.

I think I still stand by the first three paragraphs of
https://bugzilla.mozilla.org/show_bug.cgi?id=172191#c16 as what I
think its purpose ought to be. But I don't think that's the intent
of verification as currently practiced.

-David

--
𝄞 L. David Baron http://dbaron.org/ 𝄂
𝄢 Mozilla http://www.mozilla.org/ 𝄂

[Dave Mandelin]

On Thursday, August 9, 2012 4:37:11 PM UTC-7, Justin Lebar wrote:
> > That said, I'm not trying to propose something developers *must* use, rather
> > that they can use the flag if they find it useful. Think of it as more of a tool
> > you can use to cut down on unnecessary bug traffic. I'd ultimately leave it up
> > to your discretion. QA will continue to triage unflagged bugs, however flagged
> > bugs will be given priority.
>
> Okay, I appreciate that it's a request. :) Suppose I'm lazy and
> don't want to put verifyme- on all bugs which don't need verification.
> But I want to help QA out by putting verifyme- on some bugs.

It sounds like you are basically saying that we should try to automate setting of verifyme+/-, which sounds like a good idea to me. Maybe one piece of it is to use different defaults by component. In JS, I think most bugs don't need verification. The ones that do would probably be security bugs (very easy to identify automatically) and user-facing regressions. There is no reliable explicit flag for the latter, but "js:p1 in the whiteboard" is a reasonable approximation.

I'm also interested in the discussion of what verification is for and why we do it. One reason is that the better people understand the purpose of verification, the more accurately they can set the verifyme flags. For example, in the above, I'm assuming that the intent is to double-check serious bugs, but I really don't know if that's why we do it.

[Dave Mandelin]

On Thursday, August 9, 2012 4:37:11 PM UTC-7, Justin Lebar wrote:

> > That said, I'm not trying to propose something developers *must* use, rather
> > that they can use the flag if they find it useful. Think of it as more of a tool
> > you can use to cut down on unnecessary bug traffic. I'd ultimately leave it up
> > to your discretion. QA will continue to triage unflagged bugs, however flagged
> > bugs will be given priority.
>
> Okay, I appreciate that it's a request. :) Suppose I'm lazy and
> don't want to put verifyme- on all bugs which don't need verification.
> But I want to help QA out by putting verifyme- on some bugs.

[Anthony Hughes]

> It sounds like you are basically saying that we should try to automate setting of verifyme+/-, which sounds like a good idea to me.

Sure, that's a good idea, but I would think the flag needs to exist first. I've not seen any outright objections to the flag but I've not seen any endorsements of the flag either. I'm seeking developer feedback and buy-in before I file a bug, and seem to be getting a lot of good feedback but not so much buy-in.

> I'm also interested in the discussion of what verification is for and why we do it.

Great. I started a thread on this topic already (see dev-quality).

----- Original Message -----
From: "Dave Mandelin" <dman...@mozilla.com>
To: "mozilla dev planning" <mozilla.de...@googlegroups.com>
Cc: "Anthony Hughes" <ahu...@mozilla.com>, "dev-planning" <dev-pl...@lists.mozilla.org>
Sent: Thursday, August 9, 2012 5:42:11 PM
Subject: Re: Proposal for improving bug fix verification rate

[Robert Kaiser]

Anthony Hughes schrieb:

> QA triages every RESOLVED FIXED bug which is marked as status-firefoxN:fixed for a given release while in Beta.

So you're only looking at bugs that have been uplifted out-of-train and
not verifying those that have been fixed when the train was on m-c (i.e.
RESOLVED FIXED with target milestone set to that release).

Robert Kaiser

[Ehsan Akhgari]

Firstly, I think the current process of verification of bugs is not
really useful overall. I'm not saying that QA is not doing a good job
at it, but the process itself is bound to be ineffective because of the
following reasons:

1. Sometimes bugs which include an automated test case are verified.
This often includes using the testcase or the STRs for that bug, which
is what the developer has also used to create the automated test case,
which results in QA to try to replicate what happens on the test
machines which is not very useful. The review process attempts to
ensure the quality of the test case, and further verification attempts
add very little value to that.

2. Sometimes bugs don't have clear user facing risks, or we don't have
STRs for how to trigger them, and usually some time is spent for QA to
ask "what are the steps to verify this bug?" and for the developer to
answer: "code inspection to make sure the patch has landed". Not a good
use of anybody's time.

3. Sometimes bugs are verified by just repeating the STRs in comment 0.
I don't know about all developers, but most (responsible) developers
should already do this with their fix before submitting and landing the
patch, so the verification process is just duplicate effort.

4. Sometimes developers actually need QA's help to make sure a fix is
correct and works as expected. In my experience, many if not all of
those cases need QA action before the patch lands using try server
builds etc. And I have always found myself either CCing somebody from
QA on the bug or setting the qawanted keyword etc. And QA has been very
helpful in those cases, but again the verification after the fix lands
adds very little value on top of that.

Therefore, I'm going to assume that the verification process is not
useful in its current form for the rest of this post.

On 12-08-09 6:01 PM, Anthony Hughes wrote:
> To those of you who might not know me, my name is Anthony and I co-lead QA release sign-off for Firefox Desktop releases (ashughes on IRC). The QA team has been trying to find a way to better prioritize, visualize, and increase traction of bug fix verifications; especially as it pertains to tracked issues for Firefox releases. One of the biggest efficiency sinks is the back and forth that inevitably happens when we triage the hundreds of fixed bugs each release trying to decide if and how a bug fix should be verified. The following proposal outlines three initiatives we hope will improve this process.
>
> Part 1: VERIFYME Flag
> ------------------------------
> The first part of our proposal is to create a new bugzilla flag called "verifyme". The purpose would be to help us prioritize bug verification work and minimize time lost due to triaging unverifiable fixes and unnecessary verifications. The flag would replace the [qa+-?] whiteboard tags with three possible states:
>
> * verifyme+: fix is verifiable and needs to be verified
> * verifyme-: fix is unverifiable or unnecessary to verify
> * verifyme?: QA needs further instruction before verification (steps to reproduce, minimized test case, environment variables, etc)
>
> QA would routinely use these flags in our day-to-day triage but I would also encourage their use by engineers. When you mark a bug RESOLVED FIXED and you know the fix needs verification, you could set the verifyme+ flag. Alternatively, you could set the verifyme- flag when you know the FIXED bug is unverifiable or doesn't need verification. Doing so would mitigate the occurrences of QA asking how to verify or whether a bug needs verification (a common occurrence and hurdle).

This requires more work with very little gain, IMO, so I would like to
suggest that we should not proceed with using this flag.

> Part 2: VERIFYME Keyword
> -------------------------
> The second part of our proposal is for developers, engineers, managers, testers, or anyone else engaged in a bug to get used to using the "verifyme" keyword which already exists in Bugzilla. The purpose would be to indicate priority to QA. In other words, QA would triage the verifyme keyworded bugs first and foremost. We believe this will reduce the occurrence of important fixes being released untested.

That could potentially allow an official opt-in request for
verification. If this means that QA will only look at the verifyme bugs
and not all of the RESOLVED FIXED bugs, I am *very much* in favor of
this plan, as it could decrease the amount of effort in the verification
process by a huge amount and focus the efforts where it's most useful,
which would be great!

> Part 3: QA Contact Field
> ------------------------
> The third and final part of our proposal is for QA to use the QA Contact field to indicate assignment. We've already started to use the field to assign qawanted keyworded bugs to individual testers and we'd like to start using it to similarly assign responsibility for fix verification. We believe this will provide greater accessibility and visibility of QA involvement. It also creates a distinctive contributor path in that it allows volunteers to take ownership of verifying fixes.

I think that makes sense, and also it's not something that really
affects how developers work, so we shouldn't get much of a say in it
anyways! :-)

Cheers,
Ehsan

[Anthony Hughes]

Not exactly. A bug which was marked FIXED in Firefox 17 Nightly would show up in our query once Firefox 17 hits Beta.

----- Original Message -----
From: "Robert Kaiser" <ka...@kairo.at>
To: dev-pl...@lists.mozilla.org
Sent: Friday, August 10, 2012 7:51:16 AM
Subject: Re: Proposal for improving bug fix verification rate

[Jason Smith]

Per Ehsan's suggestion - Let's move this discussion over to dev.platform. I'll write up a new proposal to kick off the discussion shortly.

[Jason Smith]

On Friday, August 10, 2012 10:29:10 AM UTC-7, Anthony Hughes wrote:

[Anthony Hughes]

Excellent points Jason. Let's keep everything which relates to determining what constitutes valuable verification to my dev-platform post on the subject. We can close this thread for the time being as it doesn't sound like driving forward a verifyme bugzilla flag proposal is a good idea at this time.

Thanks

----- Original Message -----
From: "Jason Smith" <jsm...@mozilla.com>
To: dev-pl...@lists.mozilla.org
Cc: "dev platform" <dev.pl...@lists.mozilla.org>
Sent: Friday, August 10, 2012 1:37:47 PM
Subject: Re: Proposal for improving bug fix verification rate

On Friday, August 10, 2012 10:29:10 AM UTC-7, Anthony Hughes wrote:
> Not exactly. A bug which was marked FIXED in Firefox 17 Nightly would show up in our query once Firefox 17 hits Beta.
>
>
>
> ----- Original Message -----
>
> From: "Robert Kaiser" <ka...@kairo.at>
>
> To: dev-pl...@lists.mozilla.org
>
> Sent: Friday, August 10, 2012 7:51:16 AM
>
> Subject: Re: Proposal for improving bug fix verification rate
>
>
>
> Anthony Hughes schrieb:
>
> > QA triages every RESOLVED FIXED bug which is marked as status-firefoxN:fixed for a given release while in Beta.
>
>
>
> So you're only looking at bugs that have been uplifted out-of-train and
>
> not verifying those that have been fixed when the train was on m-c (i.e.
>
> RESOLVED FIXED with target milestone set to that release).
>
>
>
> Robert Kaiser
>
>
>
> _______________________________________________
>
> dev-planning mailing list
>
> dev-pl...@lists.mozilla.org
>
> https://lists.mozilla.org/listinfo/dev-planning

Hi Everyone,

Thanks a lot for the excellent feedback. This is really helping our QA team re-think our approaches to be successful at testing. Ehsan brought up a good point that this be a better suited discussion on dev.platform, so I've moved it over there. I'll move the discussion over there to continue this discussion.

I proposed an alternative approach to go about on dev.quality in alignment with the feedback specified above. Here's what I suggest we should do:

* As Ehsan stated, stick with the existing process of when explicit testing is needed on a try build by calling out qawanted and cc-ing someone from QA to ask for help in testing
* When a bug moves to resolved fixed, if more testing after the fix has landed is needed, cc someone from QA and flag qawanted to get help with formulating a test plan around the bug or implications of the bug
* Other QA team members will skim through the bugs as they come in and flag bugs as qawanted when more explicit is needed on a fixed bug other than a one-off verification.

In short, this process I'm thinking about would try to follow something similar to what the security review process, as I find their process mechanisms that they follow to be highly effective. I need to think more about the details about how our QA process could follow something similar to the security team's process, but what are some initial thoughts on that proposal?

Sincerely,
Jason Smith

[Gavin Sharp]

Robert's point is that the majority of bugs fixed on trunk are never
marked status-firefoxN:fixed - we only use that flag for bugs that are
"uplifted" (or if there's confusion about where a bug is fixed, but
that happens rarely enough).

Gavin

On Fri, Aug 10, 2012 at 10:29 AM, Anthony Hughes <ahu...@mozilla.com> wrote:
> Not exactly. A bug which was marked FIXED in Firefox 17 Nightly would show up in our query once Firefox 17 hits Beta.
>
> ----- Original Message -----
> From: "Robert Kaiser" <ka...@kairo.at>
> To: dev-pl...@lists.mozilla.org
> Sent: Friday, August 10, 2012 7:51:16 AM
> Subject: Re: Proposal for improving bug fix verification rate
>