At this moment Quality assurance is handled by a very small group of
person (between 5 and 15) and what is mostly being done is bug triage. A
few more people show up during TestDays to tests the new features. We
have a fairly nice number of people subscribed to the
Thunderbird-testers mailing list. This mailing list is used to announce
testing events so people that don't have time to follow on a daily basis
- can easily be told when a testing event is organized.
What does QA cover :
1) Bugzilla maintainance
This is where most of the work happens at the moment. This is the part
of work where bugs are being looked at. This means figuring out if a new
bug is really i) a bug, ii) not already fixed , iii) not already in
bugzilla iv) gathering as much information so developers can understand
the issue better.
There is a backlog of bugs that need to be taken care of. This is mostly
done on Bug days.
2) Testing
a) Manual
This is organized in two ways , usually on a bug day with testing of new
features. And prior to a release we use the mozilla litmus tools to run
bug days
b) Automatic
This is mostly left to devs.
Why do we need more contributions :
1) Triaging
When Thunderbird 3 hits the streets we are going to get a significant
bunch of new bugs generated by the release and people trying it out. We
will definitively need more people to read bug and triage them. Triaging
is easy but takes time - you need to figure out how bugzilla works and
what bugs are already present in it. Ideally we would need peopel to
join this effort now so they can train themselves and be ready when 3.0
is released.
2) Testing
a) Manual
Manual testing takes time and there is no way to test all the
software/hardware configuration available out there - so the more people
to join these events the better as more issues can be found.
b) Automatic
There are two things involved in automatic testing :
i) Porting existing test case from litmus and making sure they are
automated. Once automated the test will be run. And the time that was
used on running the test can be spent on another test. Doing this
automation needs manpower and people willing to spend some time on it.
ii) Adding more unit tests and writing other tests
We have the infrastructure - but not the man power. The more of those
test we have the better each releases becomes because when things break
they are discovered a lot faster.
How do we do recruitment for new QA contributors :
in bugzilla - when people are responding from time to time we ask them
if they want to join.
in #thunderbird - when helpin gsomeone when we feel they are enthusiast
about thunderbird we will engage them and ask them if they'd like to help.
What have we tried :
- Forums we use the Mozillazine forums, the mozilla news groups to
announce our event where people can show up an participate - this has
been going on for a Year or so and I think we've reached the people
involved in those medium.
- Interaction with other communities I've talked to people involved
with OpenSolaris (with results - but it's a small community), and the
major linux vendors (OpenSuse, Ubuntu and Fedora) - with less success.
The later is my fault and I need to reengage the linux distros.
Things that are planned and only work in progress :
A writing xpcshell testscript workshop - so people that are interested
in writing test, learning the internals of thunderbird and writting test
can start in a more efficient way then just starting and learning
everything by themselves.
Being more demanding when helping users in #thunderbird.
Changing the Welcome page for nightlies - To let our users know that we
need more manpower in QA.
Ludovic
--
Ludovic Hirlimann MozillaMessaging QA lead
http://www.spreadthunderbird.com/aff/79/2
What does QA cover :
2) Testing
a) Manual
in #thunderbird - when helping someone when we feel they are enthusiast
about thunderbird we will engage them and ask them if they'd like to help.
What have we tried :
- Forums we use the Mozillazine forums, the mozilla news groups to
announce our event where people can show up an participate - this has
been going on for a Year or so and I think we've reached the people
involved in those medium.
- Interaction with other communities I've talked to people involved
with OpenSolaris (with results - but it's a small community), and the
major linux vendors (OpenSuse, Ubuntu and Fedora) - with less success.
The later is my fault and I need to reengage the linux distros.
Things that are planned and only work in progress :
A writing xpcshell testscript workshop - so people that are interested
in writing test, learning the internals of thunderbird and writing test
can start in a more efficient way then just starting and learning
everything by themselves.
Being more demanding when helping users in #thunderbird.
Changing the Welcome page for nightlies - To let our users know that we
need more manpower in QA.
Any ideas or thought on what could be done to increase the community
participating to our QA effort ?
Thoughts on what we've been doing up to now ?
As you say, we have some good improvement and we are far better off now
than last year both in QA and in volunteer coders. But over time there
is an ebb and flow to our volunteer support, as in any voluntary
project. So we do need more people.
I am perhaps the least useful to comment, having been so close to the
issue. For example, established areas like mozillazine and
mozilla.support.thunderbird have talented and dedicated people there,
and yet response there has been sparse - why I don't know. So I too am
very interested in others' comments.
Big picture ... have we sought out OSS systems that are exemplary
models? (I know I haven't.) Are there such systems? How do they attract
and retain volunteers?
One would think and hope a new start page in beta builds appealing to
current has got to help. They obviously care enough to be testing. But,
how to effectively motivate them out of that comfort zone? For example,
has such a page been effective for FF? (I don't know - but I haven't
seen a steady trickle of new FF triagers over the last year. Or, perhaps
I'm just not seeing it.)
For the (existing) community ... We now have -testers mailing list and
#tb-qa which have been great. But it feels like we are ready for more
venues for the community - maybe something at QMO, a newsgroup/mailing
list, or something entirely new. (new =~ FF beta facebook
http://www.facebook.com/group.php?gid=54860359570 )
Other thoughts? New thoughts? Happy thoughts? Crazy thoughts?
refs:
- https://wiki.mozilla.org/Thunderbird:Testing
- https://wiki.mozilla.org/Thunderbird:Bugdays_Planning
I think these are great posts detailing the hard work of driving forward
with QA recruitment. It's something we struggle with across the Mozilla
Project. One thing we have to realize is that success in solving the
volunteer recruitment problem in any area, is success in every area in
terms of visibility, word of mouth, and progress. So, I'm always
thinking in terms of those kinds of grandiose visions (volunteers
everywhere!) when thinking about volunteers in the short term. That
might help to explain some of the rationale behind my thoughts. Let me
reply more specifically below....
On 4/9/09 8:35 AM, Wayne Mery wrote:
>>
>> [1] Any ideas or thought on what could be done to increase the community
>> participating to our QA effort ?
>>
yes, I'll address that.
>> [2] Thoughts on what we've been doing up to now ?
>>
I think what you're doing is good work. Unfortunately for small teams
(and even for larger teams), the hard work of recruiting means that you
have to constantly do more than you did yesterday. So what you're doing
now is excellent and good. I applaud your efforts to reach into other
OSS groups and make connections there. I think that is a good idea, and
we've talked about that before.
One thing you may wish to consider is to analyze what the impact is of
each action. Are you recruiting in everything you're doing? Often
times, a volunteer getting involved in a project is the difference
between showing up in a channel and looking at a blank screen versus
seeing someone speaking, or even better welcoming that new person.
You're trying to make a human to human connection, and that can't be
solved with fancy bots. Every single event, every single post, every
single word you say needs to invite people into your project and into
your events. I haven't attended a single Thunderbird event, so I don't
know how you're doing on this metric, but it is something to evaluate
yourselves against.
Another thing to do with your current array of tactics is to analyze
each one on the barrier to entry. Unfortunately some of our tools are
in desperate need of design revisions in order to lower their barriers
to entry (examples: Litmus and Bugzilla). Even IRC is a high barrier to
entry. I'm pushing the FFx QA folks to work to lower some of these
barriers over the next few months. That will have some residual help
for Tbird as well. But I encourage you to analyze your tactics in terms
of this metric and find places to make improvements.
>
> Big picture ... have we sought out OSS systems that are exemplary
> models? (I know I haven't.) Are there such systems? How do they attract
> and retain volunteers?
I'd love to see data on this, if anyone has it. From my experience, I
held together a volunteer calendar QA team by personal interaction. In
its heyday, the calendar-qa channel was more akin to a living room with
big comfy couches and a roaring fire where people hung out just as much
because they enjoyed the company as they did because they enjoyed the
project and the work we did. And the only way to get there is to
engage, engage, engage.
I've wondered lately if one of the reasons for the success I enjoyed
there was *because* I was also a volunteer. I can't say if that
actually had an effect, but it's interesting. I have seen quite often
in conversations that people consider the "Community" to be an amorphous
thing "out there". That's a divisive way to think of it. A much better
way to think of it is that we ARE the community. Each and every one of
us. Thinking of ourselves in the same light as the person that joined
IRC for the first time today can only do the Project a favor. I believe
that such thinking will change the interaction dynamic in a crucial way.
>
> One would think and hope a new start page in beta builds appealing to
> current has got to help. They obviously care enough to be testing. But,
> how to effectively motivate them out of that comfort zone? For example,
> has such a page been effective for FF? (I don't know - but I haven't
> seen a steady trickle of new FF triagers over the last year. Or, perhaps
> I'm just not seeing it.)
We did change the nightly start page not that long ago. Jay Patel might
have more data on that since I think he was driving it. The nightly
start page has several more "Get Involved" links and it also mentions
the QA Companion add-on. This has indeed brought a few more folks our
way, but in the numbers of tens rather than hundreds. Either we failed
to engage those folk, or they simply went their own way after their
interest waned; I can't think of a single one of them that remained
committed for any decent length of time (say a month).
>
> For the (existing) community ... We now have -testers mailing list and
> #tb-qa which have been great. But it feels like we are ready for more
> venues for the community - maybe something at QMO, a newsgroup/mailing
> list, or something entirely new. (new =~ FF beta facebook
> http://www.facebook.com/group.php?gid=54860359570 )
This is something I would suggest you improve. I tend to think that we
have begged, cajoled and pleaded for help on all our standard fora for
so long that people don't even realize that when we say, "if you'd like
to get involved, let us know," it is actually a sincere ask. As an
example, take a look at the Mozilla Labs group and what they are doing
in terms of building community:
* They have opened a part of the project that was always rather opaque
previously
* They have made it easy for (and continually try to improve the way in
which) people get involved in their projects.
* They blog constantly
* They are innovating and have a very high "shiny" index. This is going
to sound snide, and it is in no way a derision on the work of Mozilla
Labs, but being the "shiny" thing is largely a matter of message and
presentation. I mean there are limits, but those of us on the core
products can do better than we have been to date. How exciting are your
test day blog announcements, for example? I loved the way that you
changed the name of Bug Days at one point to Bug Love or some such...
There is no reason we can't do the same on QA. I'm trying to push the
FFx QA team toward thinking more out of the box and doing many of the
analyses that I mentioned above. Your project is much smaller and need
not be burdened by history or customary habits of doing things. You
should be able to push the envelope here and start thinking about
something new. Get on QMO (force that site to be application
agnostic!). Try starting up modern mailing lists (google groups, yahoo
groups?) and blogging about them. Start talking about some of the
interesting innovations (there was that timeline thing DAscher showed
once, which was cool) going on. Innovate on new ways to do QA: Hold a
Test Day using Facebook Chat as one of your mediums.
Start doing things you never thought would work online. One of the
biggest successes I had on calendar-qa was trying to do a "test case
writing day" online. And the only reason it was as big as it was is
because it was unusual, and it got picked up on Slashdot where a fierce
debate ensued about whether or not random people could write test cases.
While that argument was going on, fifty people showed up and wrote 120
test cases, of which I was able to use around 110. Many of those folks
are still involved to this day. In fact, one of them now leads the
Calendar project. :)
I think you're doing a great job in what you're doing. I think that the
biggest win for you will be to evaluate your barriers to entry and how
well all your events and interactions are recruiting for you. I hope
that you can use the nimbleness of your project to your advantage and
explore crazy ideas on getting people involved. I think the Mozilla
Project has come to an age where it must branch out and re-invent the
way it recruits if it wants to keep growing. I believe the people are
out there. I think we're not reaching them.
I hope you show those of us on the FFx side a thing or two. :)
Best,
Clint
Ok
Other thoughts
Folks I know are not really "onboard" with the current development trend, so why should they contribute their time.
They have many ideas for what they envision TB3 to be, but they think their ideas fall on deaf ears.
And I guess, when "a few folks" formulate a plan, that's to be expected.
Yeah, I know TB dev is not a Democracy, but even so, let's give a listen to the minority.
And so, I propose "Bitch Day"
We could announce this in Mozilazine forums, and the newsgroups, mozilla.support.thunderbird (and others)
Topics:
Why don't you use bugzilla
Have you tried a TB3 Beta
IRC as a way to communicate
I would be willing to try to consolidate the remarks in a post here.
You might not like what you hear, but it should be a very real "Community response"
JoeS
Clint's example of being Slashdotted is a good one, though it really
depends on luck to see if we can get Slashdotted, that in itself may
turn out beneficial, or it might just even turn out flame-able.
Somehow I do agree that IRC in itself is a high level of entry, people
want simple access to help or suggest or contribute, something along the
likes of the community helping itself out, like the SUMO
help-each-other-out-web-talk thing, might work, or might not.
That said, TB QA has come quite a ways since a year plus ago - it's
about time to consider how to move on in a scalable manner, such that it
almost becomes self-sustaining.
-Gary
nth10sd
>
> And so, I propose "Bitch Day"
>
> We could announce this in Mozilazine forums, and the newsgroups,
> mozilla.support.thunderbird (and others)
>
> Topics:
> Why don't you use bugzilla
> Have you tried a TB3 Beta
> IRC as a way to communicate
>
>
>
> I would be willing to try to consolidate the remarks in a post here.
Joe this is a great way to get feedback. Thanks for all those ideas.
Ludo
I agree that barriers to entry is a problem, at least for me. I'm a huge
Mozilla supporter and for a while I had been trying to get involved with
some QA stuff for Thunderbird, Firefox, and Lightning. I was eventually
discouraged because I didn't have privileges in Bugzilla to make
changes. I looked up how to get the permissions (which took me a great
deal of time to find) and I felt that the process used to get Bugzilla
permissions and everything was kind of discouraging. If there were other
ways to get Bugzilla privileges that didn't make it seem so exclusive,
was more clear on how to get them, and it was easier to find I think you
would be getting more non-programmers, like myself, to contribute.
Thanks,
Michael
Hi Michael. Thanks for commenting.
Perhaps you read a description that was complex, but the process is
simple and certainly not exclusive. But it is necessary - one cites a
few good examples of what they have done in bugzilla. If you have a
bugzilla account, and have commented intelligently in some bugs, chances
are very good that you already demonstrated your abilities.
Please see the description at
https://wiki.mozilla.org/Thunderbird:Bug_Triage#Canconfirm_and_Editbugs_Privileges
Do you see anything that can be improved there?
Not specific to QA but what people want is for their work to have an impact.
It's rather obvious but is worth thinking about once in a while.
One thing I'd like for more new and old QA people to do is to comment in bug
reports also if they can't reproduce. Some bugs are simple to follow the steps
to reproduce for but it takes time. That someone tried to reproduce and
a) were able to follow the steps
b) reproduced/didn't reproduce
... is fairly valuable information, and is something most people with incentive
and a nightly build are able to do to help out.
-Magnus
I confess that I have frequently gone to a bug, tried to reproduce it
and failed, then moved on without comment. My case is a little different
though, because I am usually looking for bugs that are clear enough for
me to actually fix.
rkent
> Changing the Welcome page for nightlies - To let our users know that we
> need more manpower in QA.
>
> Any ideas or thought on what could be done to increase the community
> participating to our QA effort ?
>
> Thoughts on what we've been doing up to now ?
I used to help, but I've stopped. It was way too discouraging.
Not enough bugs get fixed, and it takes way too long. On some bugs, the
only activity they've seen over the course of several years is a periodic
ping ("Is this bug still reproducible?"). It's bad enough when that
happens once. When it happens more than once ... Then there are bugs where
the duplicates have piled on and piled up, but they just live on, with no
relief in sight.
I finally decided that it was better to just stick with an old verion and
live with its known bugs than to keep trying nightlies, in hope of getting
fixes, only to constantly encounter new issues.
:(
The situation now is much better than before, the community is
revitalizing. While I would agree that the situation was indeed bad
before, Thunderbird QA is now getting back up to speed, as well as
development.
Personally, I would encourage you to "get your feet wet" again - the TB3
betas are looking to be a release with many improvements over the TB2
series. However, as with all software in the world, there will still be
bugs and issues - please do consider reporting and helping triage again.
Regards,
-Gary
nth10sd
Bugs like that should probably just be closed. That could also mean
moving it to UNCO and auto-resolving UNCO to INCOMPLETE or so after some
time without activity, as some Firefox people proposed in the thread to
change Bugzilla workflow.
Robert Kaiser
I disagree there. When I usually ping - it's because I can't reproduce
the bug - and that the reporter has been active - pinging is a good way
for me to know if the bug as been resolved by another commit.
> moving it to UNCO and auto-resolving UNCO to INCOMPLETE or so after some
> time without activity, as some Firefox people proposed in the thread to
> change Bugzilla workflow.
That's already being done - when the bug reporter does not react we use
the status whiteboard [closeme yyyy-mm-dd] and we try to have that date
match a Thursday so they end up being closed by Joshua when he does the
closeme rounds on bugdays.
That as changed over the course of the last year - Issue is the backlog
of bugs to fix is huge so getting down to that will take some time. You
of course know that patches are welcome :-)
> I finally decided that it was better to just stick with an old verion and
> live with its known bugs than to keep trying nightlies, in hope of getting
> fixes, only to constantly encounter new issues.
So what is your strategy when switching to major new versions ? You wait
for the final version to be released or you use beta's before the
version becomes final. Would help if you would start using TB3.0b3 for
instance.
Ludo
> So what is your strategy when switching to major new versions ? You wait
> for the final version to be released or you use beta's before the
> version becomes final. Would help if you would start using TB3.0b3 for
> instance.
For years, I ran trunk nightlies. I was way ahead of the betas or even
alphas. But over time, I became SO frustrated with it, and with the large
number of fundamental usability bugs that have gone unfixed for years,
and with the number of regressions (!), that I asked myself, why am I
torturing myself like this?
So, my strategy, in answer to your question is this. I am still running a
nightly trunk build from late last year. When the new release finally
releases, I will try it, and see if the regressions are still there.
(Can you "paste as quotation" from the keyboard in TB3 betas yet?
You could in all versions of TB1 and TB2.) If not, then I will stick with
that version until its dying day. By sticking with one version from the
days of "early adopter" to the days of "late relenquisher", I will minimize
the number of surprises.
These are a few of my favorite bugs/RFEs (notice the 5 digit ones)
RFE
10097 * Ability to add user to 'killfile' with a single key stroke
28211 On Send, fail to copy to Sent folder should not lose sent msg
43278 Crossposts (same Message-ID) not marked as read in other groups
86607 * UI option for disabling format=flowed
106518 "create filter" from a "To:" email address in the message pane
creates a "From" filter for the address
108650 Unread counts toggle back and forth between twisty open and
selection.
178870 * make "Run Filters on Folder" available for newsgroups
187997 Rewrap doesn't work with selected text
191881 Rewrap adds space at beginning of quoted line when previous line
is almost full length
264981 * support HTTP proxy connect feature for proxying mail/news protos
285399 Reply quotes entire paragraph as a single line
288700 * Handle delete/detach attachment feature with crypto-signed mails
297314 * Allow detach/delete of attachments in signed/encrypted messages
354272 * Need feature to delete & prevent duplicate messages in folders
383985 Order of threads in msg list pane should not be controlled by Date
Sent vs Date Received
388242 SMTP Mail Server Password dialog is modal to the wrong window
446228 rebuilding summary file doesn't fix newsgroup message counts
https://bugzilla.mozilla.org/show_bug.cgi?id=461117
--
Regards,
Peter Lairo
The browser you can trust: www.Firefox.com
Reclaim Your Inbox: www.GetThunderbird.com
Islam: http://www.jihadwatch.org/islam101/
Israel: http://www.jewishvirtuallibrary.org/jsource/myths2/
Church of the Flying Spaghetti Monster: http://www.venganza.org/
"So, why don't we ever talk about the sun's contribution to global
warming? Well, because we can't regulate it, tax it, or make it feel
guilty for what it's doing" (www.WhatYouOughtToKnow.com)
Yes it can seem repetitive and therefore may be discouraging if there
are multiple "asks" - it may not seem helpful to just ping when a new
release comes out, or when someone (newbie or not) comes on the scene
and pings without commenting as to whether they tested.
But related to Ludo's point, and not necessarily speaking directly to
Nelson's situation since 2+ years is eons ago, time/value should be an
important factor in what happens in the bug.
For example I tend to test only in cases where I think I have decent
odds of proving whether or not it's a bug or it's reproducible. (Why
take 5-20 minutes attempting to reproduce and not succeed and therefore
prove not much?) And in cases where I don't test two choices remain:
1. take a pass - not comment and hope someone else helps in the next
few months (the odds of the later happening are better now than last
year, but still not fantastic), or
2. ping reporter and see if we can't make some progress through
further reporter testing and followup
So most often I opt for #2. In 30 seconds I can write something simple,
I don't throw away the time I already invested in reading/thinking about
the bug, and in the "saved time" I can be helpful in 5+ more bugs where
I might be more productive.
Big picture ... with 40-50 (optimistically) active triagers in varying
states of activeness and tons of unique reporters per year** the greater
available manpower by far is on the reporter side. And so for that
reason alone it's reasonable to expect that many reporters will be asked
"can you reproduce doing ______."
** in 180 days ending April 10:
- 1,384 bug filers. ~1,100 filing only one bug. 275 filing 2-3 bugs. 75
4-9 bugs, 35 >10 bugs
- 2,743 new bugs filed. of those 1,291 are still open (about half
unconfirmed), 527 fixed, ~927 closed other resolutions (mostly dupes).
~300 of the 2,743 or 10% are ENH bugs
That bug fails to mention the rather simple Thunderbird workaround
mentioned in bug 192330
Can't find it. What's the comment number?
https://bugzilla.mozilla.org/show_bug.cgi?id=192330#c53
(though I thought I saw it elsewhere)