Hi all,

Many of you will recall that Alex Keybl proposed a change recently that
would have asked developers to manually verify all of their patches
after landing.  Understandably, this was met with significant concerns
regarding effective utilization of developer time, and the thread
quickly became very hard to follow.  In the very long discussion, a
heavily revised concept seemed to gain consensus, and it was suggested
that a new thread be started to discuss that revised proposal.

The proposal I'm going to link to below is intended to be
straightforward, and seeks to build upon the following three core
principles:

* For all fixed bugs, the state of test coverage should be clear and
unambiguous
* Developers have the best insight into which bugs carry the most risk
and require attention from QA
* QA is the best group to effectively test fixes, and has the best
insight into which bugs require additional testing and verification

The thrust of the proposal is simple: we should replace the current
assortment of keywords, flags, and whiteboard annotations with a single
multi-state flag (testing-status), and make setting this flag part of
the required workflow when resolving a bug in a product component as
FIXED.   Developers will be asked to assess whether a bug is either: a)
not testable, b) covered sufficiently by automated tests or c) requires
QA to assess the bug.  QA then will determine the amount and type of
testing required, and take action as appropriate.  It is intended that
this formalization of the process will provide us visibility into our
testing coverage and resource investment across our products.

The full details, including a visual workflow diagram, can be found at
https://wiki.mozilla.org/User:Mconnor/BugVerification

Any question or comments are welcome here, or you can find me on IRC.

-- Mike

https://wiki.mozilla.org/User:Mconnor/BugVerification#Visual_Workflow
looks pretty good. One thing that comes to mind, though, is most of
the time developers can determine "automated-test-needed" without help
from QA, so there should be a path to that state that doesn't involve
QA. In practice this is often "could be tested, but don't have time to
write test now". In fact I can't think of a scenario in which QAs help
would be required to determine automated-testability, and so it seems
to me like testing-needed could be renamed manual-testing-needed, and
the "QA: Is there a framework?" section of the chart could be tacked
on to the Developer "Is verification needed?" branch.

Gavin
> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning

On Mon, Jul 8, 2013 at 5:00 PM, Gavin Sharp <ga...@gavinsharp.com> wrote:
> https://wiki.mozilla.org/User:Mconnor/BugVerification#Visual_Workflow

One other comment: in our development process, I think it's most often
true that the developer is in the best position to make a judgement
about the potential utility of QA verification, but I don't think
that's ideal. They can be biased by their closeness to the problem
and/or laziness and end up making the wrong call. In an ideal world I
think a separate QA person would actually be responsible for
determining "is verification needed" (first question in the workflow
diagram), rather than the developer. But that often requires a deeper
understanding of the change that was made than QA people have (and we
don't have enough people doing QA to cover all bugs being fixed).

Given this, I think it's fine for your proposed workflow to put the
responsibility for asking that first question on the developer, but it
should probably also allow for that question to be answered by a QA
person in areas with sufficient QA resources.

Gavin

On 2013-07-08, at 8:00 PM, Gavin Sharp <ga...@gavinsharp.com> wrote:

> https://wiki.mozilla.org/User:Mconnor/BugVerification#Visual_Workflow

> looks pretty good. One thing that comes to mind, though, is most of
> the time developers can determine "automated-test-needed" without help
> from QA, so there should be a path to that state that doesn't involve
> QA. In practice this is often "could be tested, but don't have time to
> write test now". In fact I can't think of a scenario in which QAs help
> would be required to determine automated-testability, and so it seems
> to me like testing-needed could be renamed manual-testing-needed, and
> the "QA: Is there a framework?" section of the chart could be tacked
> on to the Developer "Is verification needed?" branch.

That's what I started with as well!  The reason I ended up with this particular workflow is that, in the cases where automated testing is not yet sufficient, I explicitly want QA to be looking at those bugs to assess/verify as needed.  There's probably some cases where developers will assert that manual testing cannot be useful, so they can set that value directly, but I suspect those are a relatively small minority.

-- Mike

On 2013-07-10, at 3:45 PM, Gavin Sharp <ga...@gavinsharp.com> wrote:

> On Mon, Jul 8, 2013 at 5:00 PM, Gavin Sharp <ga...@gavinsharp.com> wrote:
>> https://wiki.mozilla.org/User:Mconnor/BugVerification#Visual_Workflow
>

> One other comment: in our development process, I think it's most often
> true that the developer is in the best position to make a judgement
> about the potential utility of QA verification, but I don't think
> that's ideal. They can be biased by their closeness to the problem
> and/or laziness and end up making the wrong call. In an ideal world I
> think a separate QA person would actually be responsible for
> determining "is verification needed" (first question in the workflow
> diagram), rather than the developer. But that often requires a deeper
> understanding of the change that was made than QA people have (and we
> don't have enough people doing QA to cover all bugs being fixed).
>
> Given this, I think it's fine for your proposed workflow to put the
> responsibility for asking that first question on the developer, but it
> should probably also allow for that question to be answered by a QA
> person in areas with sufficient QA resources.

I suspect/hope that in any areas with sufficient QA resources to make these calls, those QA engineers are watching bugs and can change those values.  I believe QA should always feel empowered to push developers on quality issues where they feel there's insufficient rigour on the part of developers.

-- Mike

Could this be a role for the code reviewer? Or perhaps when submitted for
review, the author should indicate their intention for the flag and give
rationale.

(Amongst other things, this would also ensure the flag gets set!)

On Wednesday, July 10, 2013, Gavin Sharp wrote:

> On Mon, Jul 8, 2013 at 5:00 PM, Gavin Sharp <ga...@gavinsharp.com<javascript:;>>

> wrote:
> > https://wiki.mozilla.org/User:Mconnor/BugVerification#Visual_Workflow
>
> One other comment: in our development process, I think it's most often
> true that the developer is in the best position to make a judgement
> about the potential utility of QA verification, but I don't think
> that's ideal. They can be biased by their closeness to the problem
> and/or laziness and end up making the wrong call. In an ideal world I
> think a separate QA person would actually be responsible for
> determining "is verification needed" (first question in the workflow
> diagram), rather than the developer. But that often requires a deeper
> understanding of the change that was made than QA people have (and we
> don't have enough people doing QA to cover all bugs being fixed).
>
> Given this, I think it's fine for your proposed workflow to put the
> responsibility for asking that first question on the developer, but it
> should probably also allow for that question to be answered by a QA
> person in areas with sufficient QA resources.
>

> Gavin
> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org <javascript:;>
> https://lists.mozilla.org/listinfo/dev-planning
>

On 2013-07-08 8:00 PM, Gavin Sharp wrote:
> ... most of the time developers can determine "automated-test-needed"

> without help from QA, so there should be a path to that state that
> doesn't involve QA. In practice this is often "could be tested, but
> don't have time to write test now".

Closely related thing: the kind of work I do, often I find it is very
difficult for *me* to write the automated tests because I am a little
*too* familiar with the guts of the thing being tested.  (Case in point:
some of the CSS "does this parse correctly?" tests were written with
intimate knowledge of the control flow in the parser, and this actually
led to a spec bug because I didn't notice an edge case that gets lumped
together with a more common case in *our* parser, but needs special
wording in the spec.  (Handling of bad-url right before stylesheet EOF,
if you're really curious.))

So it would be nice if this process included a way for devs to request
that *someone else* write an automated, black-box test for something,
working from the spec rather than the code.  (Naturally, "sorry we don't
have the manpower" would be an acceptable response to this request.)

zw

In response to "devs to request that *someone else* write an automated,

black-box test for something, working from the spec rather than the code."

I agree with the dev requesting testing.

I would like to say that I tend to find more bugs when I look at the
spec and the code rather than just the spec alone or the code alone.

I think Zack makes a lot of good points in his statements.

Regards,
Naoki

> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org

> https://lists.mozilla.org/listinfo/dev-planning

I would highly recommend that we get QA involved in this discussion very early before this discussion goes much further. I've had several similar discussions and their resource limitations often have a significant impact on what can be done. It's important to have their insight during these early discussions.

Good to see this discussion.

Martin

----- Original Message -----

I think this is a great step forward with verifications, and I'm all for
getting rid of the confusing mess of flags and whiteboard fields we
currently use to track this stuff. Most of the time when we get
questions about automated tests that need to be written or frameworks
that are needed for such tests, those tend to come from development. I
think this is a function of the fact that development outnumbers QA 3 -4
to 1 here.  So, as long as developers can just as easily set the
"automated-test-needed"/"framework-needed" values, I think we have a
great strategy for moving forward.

I'd also like to see this folded into the review process as a means to
ensure that the flag is properly set and that the issue is discussed.

Thanks for putting this together, Mike.

On 7/11/2013 1:36 AM, Martin Best wrote:
> I would highly recommend that we get QA involved in this discussion very early before this discussion goes much further. I've had several similar discussions and their resource limitations often have a significant impact on what can be done. It's important to have their insight during these early discussions.
>
>

I've sent a note to the internal QA alias asking them to look over this
thread and post comments/questions as necessary.

Clint

I think this is a great step in the direction of simplification. We may have some bumps when we implement new process, but overall I think this is important. Thanks to all who have had the patience to put this together.

Matt Wobensmith
QA Engineer
Mozilla Corporation

Comments:

I would get rid of the immediate step of "testing-needed" and just have the developer directly set either framework-needed, automated-test-needed, or manual-test-needed directly. There will be cases where I think there will be disagreement in the case where say, a QA comes through analyzing the bugs and disagrees with the resolution. But in that case, I think the developer and QA can just work together to come to agreement when there's disagreement on the testing-status state.

QA is limited on resources, so I don't expect we'll be able to triage every single fixed bug that comes through. Each QA team does look through a subset of the fixed bugs though regularly - on Firefox OS for example, I'll usually watch the fixed blockers for a release that come in weekly.

I do think we need to clarify some of the special cases in this workflow. For example, let's say framework-needed gets set because this can be best verified through automation, but the framework isn't available. Manual testing could be done as an option on this bug as a short-term solution. How do we handle the workflow here? Should I triage and notice framework-needed is present with a comment saying "Manual testing would be helpful given that we cannot automate this in the short-term?" Should we have a workflow that moves between a one-off verification to framework-needed in this case?

One other comment:

I think there's a missing keyword in here that specifies the one-off verification status. Maybe manual-verification-needed?

I just wanted to chime in and say that I support Mike's plan, and I think the spirit of this is still in line with my original intent - to do a better job of verifying those bugs that are not already covered by automated testing.

I'm very interested to see how many bugs fall into the manual-test-needed/testing-needed and framework-needed buckets, to better understand the load being asked of QA and the A-Team.

-Alex

On Jul 8, 2013, at 11:56 AM, Mike Connor <mco...@mozilla.com> wrote:

> Hi all,
>
> Many of you will recall that Alex Keybl proposed a change recently that would have asked developers to manually verify all of their patches after landing.  Understandably, this was met with significant concerns regarding effective utilization of developer time, and the thread quickly became very hard to follow.  In the very long discussion, a heavily revised concept seemed to gain consensus, and it was suggested that a new thread be started to discuss that revised proposal.
>
> The proposal I'm going to link to below is intended to be straightforward, and seeks to build upon the following three core principles:
>
> * For all fixed bugs, the state of test coverage should be clear and unambiguous
> * Developers have the best insight into which bugs carry the most risk and require attention from QA
> * QA is the best group to effectively test fixes, and has the best insight into which bugs require additional testing and verification
>
> The thrust of the proposal is simple: we should replace the current assortment of keywords, flags, and whiteboard annotations with a single multi-state flag (testing-status), and make setting this flag part of the required workflow when resolving a bug in a product component as FIXED.   Developers will be asked to assess whether a bug is either: a) not testable, b) covered sufficiently by automated tests or c) requires QA to assess the bug.  QA then will determine the amount and type of testing required, and take action as appropriate.  It is intended that this formalization of the process will provide us visibility into our testing coverage and resource investment across our products.
>
> The full details, including a visual workflow diagram, can be found at https://wiki.mozilla.org/User:Mconnor/BugVerification
>
> Any question or comments are welcome here, or you can find me on IRC.
>
> -- Mike
>

Hi Mike,

Thanks for taking initiative on this. I fully support this proposal but I have a couple of follow up questions.

> in-automated-testsuite, automated-test-needed
How would this affect the present workflow and usage of the in-testsuite and in-qa-testsuite flags?

> manual-test-needed , in-manual-testsuite
How would this affect the present workflow and usage of the in-moztrap flag?

Thanks

Anthony Hughes
QA Engineer, Desktop Firefox
Mozilla Corporation

On 2013-07-11 5:04 PM, Anthony Hughes wrote:
> Hi Mike,
>
> Thanks for taking initiative on this. I fully support this proposal but I have a couple of follow up questions.
>
>> in-automated-testsuite, automated-test-needed
> How would this affect the present workflow and usage of the in-testsuite and in-qa-testsuite flags?
>
>> manual-test-needed , in-manual-testsuite
> How would this affect the present workflow and usage of the in-moztrap flag?

My intent is that this process would replace both of those flags, as
well as verifyme/[qa-] and anything else in common use for tracking
testing.  The idea is that there should be one canonical and consistent
status for testing.

-- Mike

On 2013-07-11 4:44 PM, jsmith...@gmail.com wrote:
> Comments:
>
> I would get rid of the immediate step of "testing-needed" and just have the developer directly set either framework-needed, automated-test-needed, or manual-test-needed directly. There will be cases where I think there will be disagreement in the case where say, a QA comes through analyzing the bugs and disagrees with the resolution. But in that case, I think the developer and QA can just work together to come to agreement when there's disagreement on the testing-status state.

The rationale behind "testing-needed" is to ensure that QA is aware of
all untested bugs landing in our products, and is able to take action as
they see fit.  If a developer just marks a bug as framework-needed, that
fix is effectively untested and unverified, which is an unmanaged risk.  
My goal here is to better manage the risk of fixes not covered by
automation.

> QA is limited on resources, so I don't expect we'll be able to triage every single fixed bug that comes through. Each QA team does look through a subset of the fixed bugs though regularly - on Firefox OS for example, I'll usually watch the fixed blockers for a release that come in weekly.

To be clear, QA would only be asked to look at the bugs that aren't
covered by tests, but are usefully testable.  Hopefully that's a
significantly smaller subset than all fixed bugs!  I understand
resources are tight, how else would you propose we identify the working
set where QA intervention is desirable?

> I do think we need to clarify some of the special cases in this workflow. For example, let's say framework-needed gets set because this can be best verified through automation, but the framework isn't available. Manual testing could be done as an option on this bug as a short-term solution. How do we handle the workflow here? Should I triage and notice framework-needed is present with a comment saying "Manual testing would be helpful given that we cannot automate this in the short-term?" Should we have a workflow that moves between a one-off verification to framework-needed in this case?

This type of case is why the proposal has a testing-needed state, rather
than a complex workflow or multiple flags.  The idea is that for the
subset of fixed bugs that are untested/unverified, QA will determine the
correct course of action.  I don't think the process should be
prescriptive for how QA makes those decisions, nor do I think we should
require any action other than that which QA deems necessary appropriate.

-- Mike

Under this proposal, do developers need to set this flag on every bug
that they fix?

Cheers,
Ehsan

On 2013-07-11, at 6:13 PM, Ehsan Akhgari <ehsan....@gmail.com> wrote:

> Under this proposal, do developers need to set this flag on every bug that they fix?

That's the expectation, yes.  I think it should be fairly easy to determine the right value for a given bug, if you've thought about how something should be tested…

-- Mike

Can we have pick a default value so that not every developer has to set
one flag per each bug they fix?  That is a huge cost, and at least most
of the bugs that I have personally worked on in the past few years will
not really benefit from this extra step at all, since they're either not
directly actionable by QA, or QA had already been looped in before the
bug is fixed.

Ehsan

On 11/07/2013 00:22, beltzner wrote:
> Could this be a role for the code reviewer? Or perhaps when submitted for
> review, the author should indicate their intention for the flag and give
> rationale.

I'm all-in for this, the code review should also include verifying the
developer did his best at writing an automated test or explaining why an
automated test can't be written or why it's not worth it.
As discussed many years ago, asking for tests should be part of the
proper review process.
If both the developer and the reviewer are responsible to evalute that,
we can reduce the possibility of miscalls.

-m

I find it a little hard to believe that setting a field value to one of three values (effectively "this is covered by tests", "this is testable but isn't", and "this is not testable") represents a huge cost to a developer.  I would expect that a developer fixing a bug already knows the answer, and if they don't, then they really _should_ know.  

Can you elaborate on why you think this is an undue burden for developers?

-- Mike

On 7/12/13 10:36 AM, Mike Connor wrote:
> Can you elaborate on why you think this is an undue burden for developers?

Mostly a cognitive burden + UI issue.  In our current workflow, and
assuming you mark when you push, not when the bug is actually marked
fixed (the latter is even more annoying), you have to open the bug, put
in a comment with the changeset ID(s) for the push, then in a different
part of the Bugzilla UI find and click the small "set flags" link, then
find the right flag in the list and set it.  It actually takes
surprising long to do this; I suggest trying it sometime on a push with
7-8 different bugs and seeing how long it takes and how frustrated you
become in the process....

Better UX would go a long way here.

-Boris

On 7/12/13 10:41 AM, Boris Zbarsky wrote:
> On 7/12/13 10:36 AM, Mike Connor wrote:
>> Can you elaborate on why you think this is an undue burden for
>> developers?
>
> Mostly a cognitive burden + UI issue.

Oh, and my description is based on my experience with trying to the
in-testsuite flag in our current world on all the bugs I check in,
because it feels like the right thing to do.  It's pretty frustrating at
times.  :(

-Boris

> Can you elaborate on why you think this is an undue burden for developers?

Well, loading the bug, scrolling to the flag, clicking on it, submitting
the form, waiting for the next page to get loaded, and then closing the
tab takes time.  It takes very little time, but it adds up and it's also
a mental burden, since the information that you're requiring us to fill
in might actually end up helping us in a few number of cases (see the
in-testsuite flag throughout the history of the project.)

Ehsan

On 2013-07-12 10:43 AM, Boris Zbarsky wrote:
> On 7/12/13 10:41 AM, Boris Zbarsky wrote:
>> On 7/12/13 10:36 AM, Mike Connor wrote:
>>> Can you elaborate on why you think this is an undue burden for
>>> developers?
>>
>> Mostly a cognitive burden + UI issue.
>
> Oh, and my description is based on my experience with trying to the
> in-testsuite flag in our current world on all the bugs I check in,
> because it feels like the right thing to do.  It's pretty frustrating at
> times.  :(

Exactly.  I also used to use in-testsuite aggressively myself, and
finally I gave up.  I honestly do not see myself adopt this proposal in
my day to day work, since it's still not clear at all what the added
value here is.

Ehsan

> Exactly.  I also used to use in-testsuite aggressively myself, and finally I gave up.  I honestly do not see myself adopt this proposal in my day to day work, since it's still not clear at all what the added value here is.

Without developer participation, it will be impossible for us to understand the true spread of unverified bugs and start devoting resources to them.

-Alex
> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning

On 2013-07-12 11:02 AM, Alex Keybl wrote:
>> Exactly.  I also used to use in-testsuite aggressively myself, and finally I gave up.  I honestly do not see myself adopt this proposal in my day to day work, since it's still not clear at all what the added value here is.
>
> Without developer participation, it will be impossible for us to understand the true spread of unverified bugs and start devoting resources to them.

This came up in the previous thread as well.  The notion of unverified
bugs is not really meaningful.  Most of our bugs will always be
unverified and that's fine, since verifying them is not useful and does
not serve any purpose.  For the subset of bugs that do require human
verification, we all agree that there should be a way to loop in qa.
This is what the old and currently abandoned verifyme keyword can be
used for, for example.

You're asking us to add a step to our day to day workflow to do
something that in most cases will provide no added value (for bugs which
do not require manual verification.)  This seems to me like something
that will be ignored by at least a number of developers.

Ehsan

If this were to be implemented, we'll need to handle the cases where a
developer has landed a patch on an integration repository, that
changeset was then merged to mozilla-central & then when mcMerge [1]
attempts to mark the bug as RESOLVED FIXED, the relevant QA flag hasn't
been set previously in the bug, so the operation fails.

In that case I would suggest mcMerge comment on the bug with a reminder
for the dev to set the flag and close the bug themselves (which will be
trivial within mcMerge) - however I wanted to call out that (in
addition to the extra bugspam,) this will delay bugs being marked as
resolved fixed - which then impacts queries used by sheriffs for
backporting fixes to branches etc. Worst case, bugs may even be left
permanently open, if devs miss the reminder.

Best wishes,

Ed

On 08 July 2013 19:56:57, Mike Connor wrote:
> Hi all,
>
> Many of you will recall that Alex Keybl proposed a change recently
> that would have asked developers to manually verify all of their
> patches after landing.  Understandably, this was met with significant
> concerns regarding effective utilization of developer time, and the
> thread quickly became very hard to follow.  In the very long
> discussion, a heavily revised concept seemed to gain consensus, and it
> was suggested that a new thread be started to discuss that revised
> proposal.
>
> The proposal I'm going to link to below is intended to be
> straightforward, and seeks to build upon the following three core
> principles:
>
> * For all fixed bugs, the state of test coverage should be clear and
> unambiguous
> * Developers have the best insight into which bugs carry the most risk
> and require attention from QA
> * QA is the best group to effectively test fixes, and has the best
> insight into which bugs require additional testing and verification
>
> The thrust of the proposal is simple: we should replace the current
> assortment of keywords, flags, and whiteboard annotations with a
> single multi-state flag (testing-status), and make setting this flag
> part of the required workflow when resolving a bug in a product
> component as FIXED.   Developers will be asked to assess whether a bug
> is either: a) not testable, b) covered sufficiently by automated tests
> or c) requires QA to assess the bug.  QA then will determine the
> amount and type of testing required, and take action as appropriate.
> It is intended that this formalization of the process will provide us
> visibility into our testing coverage and resource investment across
> our products.
>
> The full details, including a visual workflow diagram, can be found at
> https://wiki.mozilla.org/User:Mconnor/BugVerification
>
> Any question or comments are welcome here, or you can find me on IRC.
>
> -- Mike

On 12 July 2013 16:31:43, Ed Morley wrote:
> when mcMerge [1]

[1] eg https://tbpl.mozilla.org/mcmerge/?cset=9db9da2ecca0

> Most of our bugs will always be unverified and that's fine, since verifying them is not useful and does not serve any purpose.

Agreed, but there's no reason for that to be implied rather than making it explicit. I think we can agree that being able to sort these bugs into buckets with confidence, as well as having data to sift through and identify coverage holes, would be beneficial. It sounds like you're suggesting it's just not enough reward for the effort.

Is the issue the fact that this flag would need to be set at the time of landing? What if it was a field that was set when already requesting review?

-Alex

On 2013-07-12, at 12:48 PM, Alex Keybl <ake...@mozilla.com> wrote:

>> Most of our bugs will always be unverified and that's fine, since verifying them is not useful and does not serve any purpose.
>
> Agreed, but there's no reason for that to be implied rather than making it explicit. I think we can agree that being able to sort these bugs into buckets with confidence, as well as having data to sift through and identify coverage holes, would be beneficial. It sounds like you're suggesting it's just not enough reward for the effort.

I'd be interested in knowing whether that's true or not.  If 15 seconds of developer time per fixed bug gives QA clarity and focuses them on the right bugs, I think everyone wins.  In my experience, explicit always beats implicit for overall efficiency, if only for reduction of unnecessary questions around ambiguity.

> Is the issue the fact that this flag would need to be set at the time of landing? What if it was a field that was set when already requesting review?

The idea is that this flag could be set at any time, but you couldn't mark a bug fixed unless it was set.  There's certainly points in the various workflows where it'd be useful to be able to set that flag.

To Ehsan and Boris' concerns, I'm happy to stipulate that the current flag UX sucks, and I've been assuming we'd implement something that's simple and well-integrated into existing workflows. We control the horizontal and the vertical here.  If we assume it's something that's an extra click or a simple tab+arrow key action, so the mechanical overhead is minimal, does that solve most of the concerns here?

-- Mike

On 7/12/13 11:48 AM, Mike Connor wrote:
> On 2013-07-12, at 12:48 PM, Alex Keybl <ake...@mozilla.com> wrote:
>
>>> Most of our bugs will always be unverified and that's fine, since verifying them is not useful and does not serve any purpose.
>> Agreed, but there's no reason for that to be implied rather than making it explicit. I think we can agree that being able to sort these bugs into buckets with confidence, as well as having data to sift through and identify coverage holes, would be beneficial. It sounds like you're suggesting it's just not enough reward for the effort.
> I'd be interested in knowing whether that's true or not.  If 15 seconds of developer time per fixed bug gives QA clarity and focuses them on the right bugs, I think everyone wins.  In my experience, explicit always beats implicit for overall efficiency, if only for reduction of unnecessary questions around ambiguity.

Just wanted to chime in on this:
Explicitly stating which bugs would be the big gotchas would help out a
lot.  Having said that I've found bugs based on the implicitly verifying
bugs.  Unfortunately I don't think QA has the time to verify all the
bugs and in some cases, it would be most difficult to verify the bugs in
a black box style as previously mentioned.
On a similar note, sometimes a little explicit explanation in the bug
regards to the fix goes a long way for helping QA out as well. We would
<3 to have that in the bugs if you are going to ask for an explicit
verification.

>> Is the issue the fact that this flag would need to be set at the time of landing? What if it was a field that was set when already requesting review?
> The idea is that this flag could be set at any time, but you couldn't mark a bug fixed unless it was set.  There's certainly points in the various workflows where it'd be useful to be able to set that flag.
>
> To Ehsan and Boris' concerns, I'm happy to stipulate that the current flag UX sucks, and I've been assuming we'd implement something that's simple and well-integrated into existing workflows. We control the horizontal and the vertical here.  If we assume it's something that's an extra click or a simple tab+arrow key action, so the mechanical overhead is minimal, does that solve most of the concerns here?
>
> -- Mike

On 2013-07-12 12:48 PM, Alex Keybl wrote:
>> Most of our bugs will always be unverified and that's fine, since verifying them is not useful and does not serve any purpose.
>
> Agreed, but there's no reason for that to be implied rather than making it explicit. I think we can agree that being able to sort these bugs into buckets with confidence, as well as having data to sift through and identify coverage holes, would be beneficial. It sounds like you're suggesting it's just not enough reward for the effort.

There is a reason.  Making that explicit takes time and energy and
doesn't generate any additional information.  You have a boolean flag,
I'm advocating to defaulting it to false, you're advocating to make it
not have a default value.  The latter is more work for everybody, with
the exact same result.

> Is the issue the fact that this flag would need to be set at the time of landing? What if it was a field that was set when already requesting review?
>

On 2013-07-12 2:48 PM, Mike Connor wrote:

>
> On 2013-07-12, at 12:48 PM, Alex Keybl <ake...@mozilla.com> wrote:
>
>>> Most of our bugs will always be unverified and that's fine, since verifying them is not useful and does not serve any purpose.
>>
>> Agreed, but there's no reason for that to be implied rather than making it explicit. I think we can agree that being able to sort these bugs into buckets with confidence, as well as having data to sift through and identify coverage holes, would be beneficial. It sounds like you're suggesting it's just not enough reward for the effort.
>

> I'd be interested in knowing whether that's true or not.  If 15 seconds of developer time per fixed bug gives QA clarity and focuses them on the right bugs, I think everyone wins.  In my experience, explicit always beats implicit for overall efficiency, if only for reduction of unnecessary questions around ambiguity.

It will definitely take more than 15 seconds, since it's a Bugzilla
interaction.  ;-)  But seriously, I think it's perhaps more accurate to
maybe say 1 minute to include the context switch (which varies depending
on the person.)  To put that into numbers, for me personally over the
past year, that's 501 fixed bugs times 1 minute, which is more than a
full working day.  So, that's one developer's day worth of time to
generate no additional data.

>> Is the issue the fact that this flag would need to be set at the time of landing? What if it was a field that was set when already requesting review?
>

> The idea is that this flag could be set at any time, but you couldn't mark a bug fixed unless it was set.  There's certainly points in the various workflows where it'd be useful to be able to set that flag.
>
> To Ehsan and Boris' concerns, I'm happy to stipulate that the current flag UX sucks, and I've been assuming we'd implement something that's simple and well-integrated into existing workflows. We control the horizontal and the vertical here.  If we assume it's something that's an extra click or a simple tab+arrow key action, so the mechanical overhead is minimal, does that solve most of the concerns here?

My concern is different than Boris'.  I am arguing that storing this
information explicitly doesn't generate additional data than storing it
implicitly and count the absense of the flag as test-not-needed.

Ehsan

On 2013-07-12 3:03 PM, "平田修樹 (Naoki Hirata)" wrote:

> On 7/12/13 11:48 AM, Mike Connor wrote:
>> On 2013-07-12, at 12:48 PM, Alex Keybl <ake...@mozilla.com> wrote:
>>
>>>> Most of our bugs will always be unverified and that's fine, since
>>>> verifying them is not useful and does not serve any purpose.
>>> Agreed, but there's no reason for that to be implied rather than
>>> making it explicit. I think we can agree that being able to sort
>>> these bugs into buckets with confidence, as well as having data to
>>> sift through and identify coverage holes, would be beneficial. It
>>> sounds like you're suggesting it's just not enough reward for the
>>> effort.
>> I'd be interested in knowing whether that's true or not.  If 15
>> seconds of developer time per fixed bug gives QA clarity and focuses
>> them on the right bugs, I think everyone wins.  In my experience,
>> explicit always beats implicit for overall efficiency, if only for
>> reduction of unnecessary questions around ambiguity.

> Just wanted to chime in on this:
> Explicitly stating which bugs would be the big gotchas would help out a
> lot.

Yes, nobody is disputing that.

 > Having said that I've found bugs based on the implicitly verifying
> bugs.

That's true.  But I think we should work on the assumption that the code
that developers write is going to have bugs, so if you spend enough
energy in verifying a given bug's fix, you're bound to find a bug.  The
question is, where should the attention be focused.

 > Unfortunately I don't think QA has the time to verify all the
> bugs and in some cases, it would be most difficult to verify the bugs in
> a black box style as previously mentioned.
> On a similar note, sometimes a little explicit explanation in the bug
> regards to the fix goes a long way for helping QA out as well. We would
> <3 to have that in the bugs if you are going to ask for an explicit
> verification.

Agreed.

Please note that the thing that I'm questioning is whether it's valuable
to spend time to explicitly mark the items that are *not* going to be
useful to QA.

Ehsan

>>> Is the issue the fact that this flag would need to be set at the time
>>> of landing? What if it was a field that was set when already
>>> requesting review?
>> The idea is that this flag could be set at any time, but you couldn't
>> mark a bug fixed unless it was set.  There's certainly points in the
>> various workflows where it'd be useful to be able to set that flag.
>>
>> To Ehsan and Boris' concerns, I'm happy to stipulate that the current
>> flag UX sucks, and I've been assuming we'd implement something that's
>> simple and well-integrated into existing workflows. We control the
>> horizontal and the vertical here.  If we assume it's something that's
>> an extra click or a simple tab+arrow key action, so the mechanical
>> overhead is minimal, does that solve most of the concerns here?
>>

The value of indicating something up-front that doesn't need QA involvement vs. assuming no QA involvement is that it provides a data point as a triage value to what QA work has been done vs. what's additionally needed. The problem of assuming no modification = no QA involvement is that creates a confusion point over two concepts - something that's untriaged vs. something that has been triaged.

For QA involvement, we usually end up burning down that a subset of bugs as a result with nothing indicated on the bug and end up adding the verifyme keyword to what we think is necessary as a result. If we had bugs flagged down immediately indicating no QA involvement necessary, then that can streamline our triage process to:

1. Skim the list of bugs that have been marked no QA involvement
2. Double check we agree with it

Then that automatically filters out a list of bugs we know have been confirmed by both parties (developer & QA) that do not need QA involvement, so nobody has to chase down to find out information if QA involvement is needed on X bug. For important bugs per release (e.g. tracking bugs on Desktop/Android, blocker bugs for Firefox OS), this would greatly streamline an analysis process to know where help isn't needed based on the developer's viewpoint.

I'm not sure I understand your arguments though on why it's a lot overhead to add a flag on check-in to indicate testing-status - can we figure out ways to streamline that process?

On 2013-07-12 5:08 PM, jsmith...@gmail.com wrote:
> The value of indicating something up-front that doesn't need QA involvement vs. assuming no QA involvement is that it provides a data point as a triage value to what QA work has been done vs. what's additionally needed. The problem of assuming no modification = no QA involvement is that creates a confusion point over two concepts - something that's untriaged vs. something that has been triaged.

Hmm, so there is a triage process that happens?  Does that happen over
the entire set of bugs in all (engineering) components?

I guess I'm a bit puzzled now, can you please describe what the goal of
this triage is?  I don't want to make the wrong guesses.

> For QA involvement, we usually end up burning down that a subset of bugs as a result with nothing indicated on the bug and end up adding the verifyme keyword to what we think is necessary as a result. If we had bugs flagged down immediately indicating no QA involvement necessary, then that can streamline our triage process to:
>
> 1. Skim the list of bugs that have been marked no QA involvement
> 2. Double check we agree with it

Hmm, I'll let you clarify the triage process more, but with this, seems
like you can do the triage just as fine if you take the assumption of
"no keyword" == "no QA verification needed".

> Then that automatically filters out a list of bugs we know have been confirmed by both parties (developer & QA) that do not need QA involvement, so nobody has to chase down to find out information if QA involvement is needed on X bug. For important bugs per release (e.g. tracking bugs on Desktop/Android, blocker bugs for Firefox OS), this would greatly streamline an analysis process to know where help isn't needed based on the developer's viewpoint.
>
> I'm not sure I understand your arguments though on why it's a lot overhead to add a flag on check-in to indicate testing-status - can we figure out ways to streamline that process?

Perhaps I'm misunderstanding the goals here.  Hopefully you can help
clear things up on what this proposed flag is actually going to be used for.

Thanks!
Ehsan

On 07/08/2013 11:56 AM, Mike Connor wrote:
> The thrust of the proposal is simple: we should replace the current
> assortment of keywords, flags, and whiteboard annotations with a
> single multi-state flag (testing-status), and make setting this flag
> part of the required workflow when resolving a bug in a product
> component as FIXED.   Developers will be asked to assess whether a bug
> is either: a) not testable, b) covered sufficiently by automated tests
> or c) requires QA to assess the bug.  QA then will determine the
> amount and type of testing required, and take action as appropriate.  
> It is intended that this formalization of the process will provide us
> visibility into our testing coverage and resource investment across
> our products.

My perspective as a developer who mainly works within js/src and who has
an allergy to using the bugzilla web UI for standard bug workflow:

First, I would prefer that this flag be set when (or just before)
requesting review. The main benefit I see in this process is reminding
the developer to explicitly consider what sort of testing is
appropriate. It's a good time to realize that an automated test should
be written, which means I'll go ahead and write one rather than setting
this flag to some suboptimal value. If the flag is set when landing, or
worse, when the bug is resolved, then it's too late -- I already have
r+, I'm mentally done with the bug unless it crash lands or something,
there's no way I'm going to go back at that point and write a test.
Especially if I'm the sort who feels the need to request review on
tests, and I usually am.

Second, when scanning through the last several patches I have landed, I
find myself wanting another status value: no-test-needed. It is
different from not-testable, because I could imagine a convoluted test
scenario that would test my change, but it would be a waste of my time
to write it and a waste of everyone else's time to run it over and over
as part of an automated suite. This is for things like refactoring or
mostly-refactoring patches, minor fixes, debug-only or depending on
nonstandard config flags or otherwise NPOTB patches, etc. When I scanned
through my commit history, I found a lot of these. I'd probably want
this as the default answer for most of the simple stuff, since
not-testable is often a lie but that doesn't mean it's worth writing a
test for.

Third, there's no way I'd use this if it required me going to the web
UI. I don't do that now for most bugs. In fact, a very common workflow
for me is to write the patch first, then file the bug with bzexport
--new (requesting review with -r at the same time), then reading the
review results in bugmail, then pushing, then using my sekrit |hg
pushed| command to comment with the landed revsets. If you look, you'll
count exactly zero interactions with the bugzilla UI. When attaching a
patch to a bug that somebody else filed, I'll probably go through the UI
once to see what the problem is, then do bzexport and continue through
the workflow above for the rest. (Don't get me wrong; I use the bugzilla
UI all the time for other people's bugs, and when discussion breaks out
on my bugs.)

...which basically means I won't set this flag unless bzexport supports
it. Which is fine, I can implement it there. But at least for myself,
I'd probably default it to no-test-needed (if available, otherwise...
not-testable, maybe? And ignore the lie?) Or maybe even add in some
cleverness so that bzexport would say "if testing-status is unset or
no-test-needed, and there's a test in this patch, set
testing-status=in-automated-testsuite." (The no-test-needed part is
because I'll often put the tests in a separate attachment, and this
logic is applied per-patch.)

And of course, if the flag is set to no-test-needed, then the reviewers
should decide whether they agree. But I'd like to point out that that
isn't straightforward, since a reviewer might click on the 'review' link
in bugmail and only see that one patch, not the flags on the bug. Moar
tooling fixes needed.

>
> The full details, including a visual workflow diagram, can be found at
> https://wiki.mozilla.org/User:Mconnor/BugVerification

That's an excellent way of describing the proposal, btw. Much better
than a wall-o-text post (or wading through a whole thread of them.) In
fact, if I squint, it kinda looks like a bikeshed...

Come to think of it, how about <http://hotpink.bikeshed.com/> -- just
rename not-testable to no-test-needed (or no-testing-needed)? It doesn't
seem that useful to explicitly identify bugs that cannot be tested. I
mean, it's theoretically interesting to collect lists of bugs for which
an adequate test harness is impractical or ridiculous, or bugs with no
observable effect on functionality (eg pure performance optimizations).
But not worth the developer overhead. Maybe that's what everyone has
been assuming? It looks like Ehsan is, given "...count the absense of
the flag as test-not-needed."

Ehsan's proposal of making unset == no-test-needed makes me a little
nervous, since it conflates no-test-needed with
never-noticed-this-flag-before-is-it-new and
i-havent-thought-about-testability-yet-maybe-ill-get-to-that-later, but
that doesn't seem like a huge deal to me so I'd be fine either way.

That's a good point Steve. I like the idea of setting it when you ask
for review -- it prevents developers from needing to head back to the
bug after getting review in order to set the flag before landing, and we
can wire it into bzexport.  We could also have the flag show up on the
review panes so that it's obvious when you're reviewing  a bug what the
flag is set to.  That's some amount of work on the bugzilla side, but I
don't think it's a ton. (I'll have to follow up with my bugzilla gurus
to find out just how much).

Thanks,

Clint

On 2013-07-14 4:56 AM, Steve Fink wrote:
> First, I would prefer that this flag be set when (or just before)
> requesting review. The main benefit I see in this process is reminding
> the developer to explicitly consider what sort of testing is
> appropriate. It's a good time to realize that an automated test should
> be written, which means I'll go ahead and write one rather than setting
> this flag to some suboptimal value. If the flag is set when landing, or
> worse, when the bug is resolved, then it's too late -- I already have
> r+, I'm mentally done with the bug unless it crash lands or something,
> there's no way I'm going to go back at that point and write a test.
> Especially if I'm the sort who feels the need to request review on
> tests, and I usually am.

I am not convinced that we should force every developer to mess with
this flag on every patch they upload just as an educational tool for
folks who do not think about testing their patches as part of their work
flow.  Especially in cases where the patch already includes an automated
test...  Why should you need to set a flag saying "hey, as you can see,
my patch includes a test"?

> Come to think of it, how about <http://hotpink.bikeshed.com/> -- just
> rename not-testable to no-test-needed (or no-testing-needed)? It doesn't
> seem that useful to explicitly identify bugs that cannot be tested. I
> mean, it's theoretically interesting to collect lists of bugs for which
> an adequate test harness is impractical or ridiculous, or bugs with no
> observable effect on functionality (eg pure performance optimizations).
> But not worth the developer overhead. Maybe that's what everyone has
> been assuming? It looks like Ehsan is, given "...count the absense of
> the flag as test-not-needed."

I think we're talking about two issues here:

1. What we can do to communicate to QA that a bug needs manual
verification.  mconnor's proposal suggests we should add a flag to every
bug for this.  My suggestion is that we should use an existing opt-in
facility (such as the qawanted keyword) that will only be used for bugs
which actually need QA attention.  I don't care much about the vehicle
in which we deliver this information.  My objection is mostly focused on
the opt-out semantics of the current proposal.

2. How to consolidate different flags regarding testing.  #2 sort of
depends on #1, but I don't expect such a consolidation to be a huge
advantage over the in-testsuite flag, which some people (myself
included) have used occasionally in the past, but others have
categorically ignored.  I don't expect adding a new way of basically
doing the same thing make people actually use this flag in the long run.

> Ehsan's proposal of making unset == no-test-needed makes me a little
> nervous, since it conflates no-test-needed with
> never-noticed-this-flag-before-is-it-new and
> i-havent-thought-about-testability-yet-maybe-ill-get-to-that-later, but
> that doesn't seem like a huge deal to me so I'd be fine either way.

You need to solve the problem of educating people regardless of what new
proposal, if any, is adopted.

Cheers,
Ehsan

----- Original Message -----
> From: "Mike Connor" <mco...@mozilla.com>
>
> The full details, including a visual workflow diagram, can be found at
> https://wiki.mozilla.org/User:Mconnor/BugVerification

I apologize for getting back to this late, I'm just getting back from a holiday. I just want to add that explicitly setting "automated test needed" and "manual test needed" states represents a huge opportunity for community involvement. Being able to automatically present a list of stuff to try on Nightly or offer to mentor people through the test-building process would be a great onramp for potential contributors.

Thanks,

- mhoye

Comments highlighted by quoted areas:

"Hmm, so there is a triage process that happens?  Does that happen over the entire set of bugs in all (engineering) components?"

The desktop QA team I know usually looks over the tracking-firefoxN bugs to determine if verification is needed on the bug or not. For Firefox OS, a feature QA owner might also triage incoming bugs for verification candidates. For example, I'll triage Gaia Calendar bugs to determine if there's value in doing bug verification. This style of triage however does not happen with every single bug - it's usually a subset of bugs that might fall into such buckets as:

* Targeted a for a Firefox release (tracking-firefox24+)
* Blocking 1.1 release of Firefox OS

There are definitely components in bugzilla where I think the triage process would be less prominent vs. others where I think the triage process would be more prominent as a result though.

"I guess I'm a bit puzzled now, can you please describe what the goal of this triage is?  I don't want to make the wrong guesses."

From a feature owner QA perspective, I'd say it's to identify exploratory test points from patches that land for high impact bugs. So what bugs I'm looking to find include such things as:

* High regression risk bugs or patches in high regression risk areas
* Partner/Certification impacting bugs
* Smoketest regression bugs
* Patches with high complexity
* sec-high or sec-critical bugs

This tends to be more critical in the Firefox OS QA process given the current regression rate that's present and the potential partner impact of regressions that occur from patches that land.

"Hmm, I'll let you clarify the triage process more, but with this, seems like you can do the triage just as fine if you take the assumption of "no keyword" == "no QA verification needed"."

The problem with saying no keyword means no QA verification needed is that there are many bugs right now that go unmarked without a verification flag that I think definitely could get value from getting additional testing. I think that's one of the key motiviations for why the thread before this started this proposal came about - especially in the Firefox OS context.

FWIW - I did talk to a few other QAs about your concerns you are raising. I think I have an idea of a compromise that could address your concerns below.

Each bugzilla component comes with a default testing-status value that is set on landing by default. For components where there's commonly a non-testable area manually, we could default to non-testable. If the components commonly benefit from having automated test coverage, we could default to automated-test-needed. Same idea for manual-verification-needed and manual-test-needed status values. For special cases where the choice is different than the default, the developer landing the patch could set the appropriate value. If there's disagreement on the appropriate value, then the developer and respective QA can workout what the appropriate testing-status value should be.

With the above approach, I think that can help streamline the workflow such that developers do not have to do extra work on landing unless the special case workflow is needed. Does that work?

On 2013-07-15 6:36 PM, jsmith...@gmail.com wrote:
> Comments highlighted by quoted areas:
>
> "Hmm, so there is a triage process that happens?  Does that happen over the entire set of bugs in all (engineering) components?"
>
> The desktop QA team I know usually looks over the tracking-firefoxN bugs to determine if verification is needed on the bug or not. For Firefox OS, a feature QA owner might also triage incoming bugs for verification candidates. For example, I'll triage Gaia Calendar bugs to determine if there's value in doing bug verification. This style of triage however does not happen with every single bug - it's usually a subset of bugs that might fall into such buckets as:
>
> * Targeted a for a Firefox release (tracking-firefox24+)
> * Blocking 1.1 release of Firefox OS
>
> There are definitely components in bugzilla where I think the triage process would be less prominent vs. others where I think the triage process would be more prominent as a result though.

I see.  So the first thing here is that we should limit the scope of
this proposal (and any alternatives) to cover only those bugs.  These
bugs are a minority of all of the bugs that people work on.

> "I guess I'm a bit puzzled now, can you please describe what the goal of this triage is?  I don't want to make the wrong guesses."
>
>  From a feature owner QA perspective, I'd say it's to identify exploratory test points from patches that land for high impact bugs. So what bugs I'm looking to find include such things as:
>
> * High regression risk bugs or patches in high regression risk areas
> * Partner/Certification impacting bugs
> * Smoketest regression bugs
> * Patches with high complexity
> * sec-high or sec-critical bugs

Understood.  This makes sense.  I believe that you can find this subset
through querying over other indicators in a bug (for example to find
sec-high/crit bugs) and in some cases such as partner impacting bugs,
the flag can be set on the bug asking for QA attention by the developer.

> This tends to be more critical in the Firefox OS QA process given the current regression rate that's present and the potential partner impact of regressions that occur from patches that land.

Yeah.  This point was brought up in the thread that Alex started on this
subject a while ago.  The testing situation in Firefox OS is currently
so different than desktop/Android that it might make sense to impose
different rules for Firefox OS bugs.  But I don't believe that the core
concerns apply to many other bugzilla components, and in fact in my
experience some of our core components at least have never gotten QA
action (unless somebody explicitly asked for it, of course.)

> "Hmm, I'll let you clarify the triage process more, but with this, seems like you can do the triage just as fine if you take the assumption of "no keyword" == "no QA verification needed"."
>
> The problem with saying no keyword means no QA verification needed is that there are many bugs right now that go unmarked without a verification flag that I think definitely could get value from getting additional testing. I think that's one of the key motiviations for why the thread before this started this proposal came about - especially in the Firefox OS context.

Well, I agree that having a huge backlog of bugs before this new policy
is a problem, no doubt about that.  But I don't think that increasing
the amount of needless work that people must do indefinitely is the
right solution.  For existing bugs, why can't the module owner just
triage the list and provide QA with a subset of those bugs which require
action from QA?

> FWIW - I did talk to a few other QAs about your concerns you are raising. I think I have an idea of a compromise that could address your concerns below.
>
> Each bugzilla component comes with a default testing-status value that is set on landing by default. For components where there's commonly a non-testable area manually, we could default to non-testable. If the components commonly benefit from having automated test coverage, we could default to automated-test-needed. Same idea for manual-verification-needed and manual-test-needed status values. For special cases where the choice is different than the default, the developer landing the patch could set the appropriate value. If there's disagreement on the appropriate value, then the developer and respective QA can workout what the appropriate testing-status value should be.
>
> With the above approach, I think that can help streamline the workflow such that developers do not have to do extra work on landing unless the special case workflow is needed. Does that work?

Thanks a lot for discussing this with QA folks!  I think doing that
makes sense for some components, but for some other components it may
not be very easy to decide what that default value should be.  Basically
there are Bugzilla components which cover a very small piece of our
code, and there are other components which cover a huge amount of code
(see Core::DOM as an example for the latter.)  I'm not sure if this
would make sense for the latter group.

That said, if we restrict this to Firefox OS components, then I won't
have any further concerns!

Cheers,
Ehsan

> why can't the module owner just triage the list and provide QA with a subset
> of those bugs which require action from QA?

Good question. I think there are some who would have that be QA's job. It is my belief that Module Owners are in a position to advise QA on high-risk changes and QA is in a position to advise Module Owners on testing approaches and quality concerns. I see no reason why we shouldn't be collaborating more and the above workflow is an excellent example of how we can achieve that, together.

Anthony Hughes
QA Engineer, Desktop Firefox
Mozilla Corporation