Organizing Lightning QA
Clint Talbert
1. Covering (testing) the code thoroughly
2. Not regressing
3. Finding/addressing defects efficiently
4. Have a good idea of the health and stability of the codebase
I'm sure you can all think of more reasons why it will help everyone
from end users to developers to QA folk.
The big question is how to accomplish this, and that's what I need ideas
and help to solve.
There are several areas that need to be addressed:
1. Regression testing
2. UI smoketests
3. Bug triage and assignment
4. Bugzilla component reorganization
5. Test coordination and planning (between people)
Considering the state of the project right now, I think we should focus
on Regression testing, bugzilla components, and Bug triage.
We need Regression testing in order to test the features currently
provided by the back-end as well as the new features that will be
landing in the codebase very soon. We need a regression test strategy,
some type of test suite that can be run for each build (or proposed
patch) that will ensure the new features work and that they do not break
existing functionality.
Ideally, these regressions would be so automated that tinderbox could
run them when it finishes a build. Dan has already begun down this path
with the items at: http://lxr.mozilla.org/seamonkey/source/calendar/test/
The bugzilla components need reorganization. That's a given. Several
proposals are outlined at:
http://wiki.mozilla.org/Calendar:Bugzilla_Components. I'm in favor of a
functional/feature driven component list. I think components should be
specific but also easy enough to understand so that uninvolved people
can still correctly submit bugs without too much guessing.
And finally, we need to address Bug triage. This goes along with the
new features and the 1.0 roadmap. We will need to have a method to
assign various people responsibility for components, then those folks
will only need to triage the bugs for that component, and they will not
have to monitor the entire list. We also need to go through the old
defects and find the ones that still apply to the current product. This
will be vital as the testing effort increases.
Clint
Simon Paquet
> Organizing the Lightning QA effort will aid us with:
Can we add Sunbird here. Since we have two main apps and both of them
benefit from round about 98% of our code changes, this seems a
reasonable thing to do.
> The bugzilla components need reorganization. That's a given. Several
> proposals are outlined at:
> http://wiki.mozilla.org/Calendar:Bugzilla_Components. I'm in favor
> of a functional/feature driven component list. I think components
> should be specific but also easy enough to understand so that
> uninvolved people can still correctly submit bugs without too much
> guessing.
Agreed. IMO the bugzilla component reorg has stalled long enough and
should be targeted right after Sunbird 0.3a2 is released.
> And finally, we need to address Bug triage. This goes along with
> the new features and the 1.0 roadmap. We will need to have a method
> to assign various people responsibility for components, then those
> folks will only need to triage the bugs for that component, and
> they will not have to monitor the entire list. We also need to go
> through the old defects and find the ones that still apply to the
> current product. This will be vital as the testing effort increases.
Since I have around 3,5 years experience in Mozilla-related QA work,
let me say that assigning components to people is a thing that has
failed miserably in nearly all areas of Mozilla code. The only
exception being people being paid to do Mozilla QA work.
In the "good old times", the QA Contact field in Bugzilla was assigned
to a single person and that person was basically responsible for all
the QA work in that area. This lead to the impression that nobody else
was required to do QA work there and people headed off to other
adventures. Also not all components are equal of size so one person
might get a pretty huge component (like Sunbird Front-end) and another
person might get a component like Storage provider, which is rather
small.
To get around this, the pseudo-mailaccounts were introduced, which
everyone interested in doing QA could watch and do the QA that was
necessary. At least in the last 6-9 months that has worked pretty
well. Our goal should just be to broaden the base of QA volunteers
and help them to get started.
This includes giving them appropriate Bugzilla privileges (I can do
that) and prepare texts to help them get started. Most (if not all)
of these texts probably exists as general Mozilla QA texts already
and can be used verbatim or with just some minor tweaks.
--
Simon Paquet
Sunbird/Lightning/Calendar website maintainer
http://www.mozilla.org/projects/calendar
Clint Talbert
Simon Paquet wrote:
> Clint Talbert wrote on 05. Apr 2006:
>
>> Organizing the Lightning QA effort will aid us with:
>
>
> Can we add Sunbird here. Since we have two main apps and both of them
> benefit from round about 98% of our code changes, this seems a
> reasonable thing to do.
>
> ..snip..To get around this, the pseudo-mailaccounts were introduced, which
> everyone interested in doing QA could watch and do the QA that was
> necessary. At least in the last 6-9 months that has worked pretty
> well.
Excellent ideas. I jumped into this without very much mozilla-specific
QA experience, so your knowledge of what has worked and has not worked
is invaluable.
Dan Mosedale
>
> Since I have around 3,5 years experience in Mozilla-related QA work,
> let me say that assigning components to people is a thing that has
> failed miserably in nearly all areas of Mozilla code. The only
> exception being people being paid to do Mozilla QA work.
>
> In the "good old times", the QA Contact field in Bugzilla was assigned
> to a single person and that person was basically responsible for all
> the QA work in that area. This lead to the impression that nobody else
> was required to do QA work there and people headed off to other
> adventures. Also not all components are equal of size so one person
> might get a pretty huge component (like Sunbird Front-end) and another
> person might get a component like Storage provider, which is rather
> small.
>
> To get around this, the pseudo-mailaccounts were introduced, which
> everyone interested in doing QA could watch and do the QA that was
> necessary. At least in the last 6-9 months that has worked pretty
> well. Our goal should just be to broaden the base of QA volunteers
> and help them to get started.
OK, I'll withdraw my QA ownership proposal that I made in the Bugzilla
component re-org thread. One thing I think that would be very helpful
would be if we could document how we prioritize bugs in general. In
particular, dataloss bugs are obviously very-high priority, but another
class of bugs that I think should be dealt with fairly uniformly and at
a fairly high priority is incorrect display of data.
Dan