Results of Community Research
Max Kanat-Alexander
former contributors why they left. Also, as you may not know, before
that, for several days I was involved in a statistical analysis of the
history of Bugzilla contributions. Basically, I looked at how many
different contributors we had in CVS for every 30 day period since the
start of the Bugzilla Project. I wrote a script that gave me the data by
parsing out who the "Patch By" email address was, or I used the
committer if there was no "Patch By" in the commit message, and I joined
people who I knew to be duplicates, which were very few.
The resulting graph was very interesting. It showed two interesting things:
* Freezes kill the community--every time we had a code freeze, the
number of contributors would dramatically drop and then recover slowly.
* Despite this, the size of the community generally increased until the
start of 2005, and then leveled off through 2005, and then started to
slowly decrease after that. No matter how I reorganized the data, or how
I analyzed it, 2005 was the critical year.
So my question was--what happened in 2005 that changed things so
dramatically? After QUITE a bit of searching, I discovered the following:
1) The "review" and "review-requests" mailing lists were split at the
start of 2005. Before, all reviewers had at least seen every single
review request. Now they had to subscribe to review-requests and
actually *request* all that spam, which nobody was going to do. (I
didn't even know about the review-requests list by the time I became a
reviewer, which was after this point.) I think that this particularly
hurt people's ability to get review "from the wind", since only old-time
and very dedicated reviewers would have been on the list where those
requests went.
2) We froze *twice* for 2.20.
#2 wouldn't have been so bad, and we would have recovered, I think, if
not for #1. But why would that review-requests change have such a
profound effect? Well, the answers to that were at least partially in
the survey results. (By the way, I really appreciate all the answers
that I got, they were extremely helpful.)
The #1 reason that people leave is simply that they no longer have time
to contribute, or that they were contributing as part of their job and
now they have changed jobs. The survey data showed that it is to some
degree inevitable that contributors are going to leave.
I'd like to note here that nobody gave "the review process is too hard"
as a reason for leaving. Some people here and there have banged the drum
for a lowering of quality as an incentive to attract and retain
community members, and there was absolutely no evidence in the survey
that that would improve the situation at all.
Both of the former documentors who responded mentioned the lack of
kudos from the community as one reason that they weren't motivated to
stay in the community. Several other people mentioned negative
interactions or simply a lack of appreciation as part of the reason that
they left.
In fact, for nearly every person who responded, the factors involved in
not staying, beyond "my job changed" or "I didn't have time", were
surprisingly personal. I know that we all work with computers, and
perhaps we'd like to think that engineering should be a totally cold
scientific profession where we all do our jobs correctly according to
the requirements of the machine, and not worry about our emotional or
personal involvements. But nothing could be further from the truth--the
personal interactions that people have with community members, the
amount they feel appreciated, and the amount they feel assaulted, are
actually the MOST IMPORTANT ASPECTS OF RETAINING COMMUNITY MEMBERS.
And *retaining* community members is what we have to do, particularly
*new* community members. Because the statistical analysis and the survey
both showed that the eventual leaving of many contributors is
*inevitable*. What we have to focus on is locating, accepting, and
encouraging *new* contributors. If we don't continuously *grow* the
community with new members, it will continuously shrink as old ones
leave, no matter what else we do.
The survey tells me that the keys to retaining new community members are:
1) Extremely rapid response to review requests. No matter how much
review a contributor has to go through, they usually will go through it
as long as there aren't LONG DELAYS between the request and the response.
2) Extreme kindness and visible appreciation. When people contribute on
a volunteer basis, they aren't getting paid in money, they are getting
paid in admiration, appreciation, the sense of a job well done, and the
knowledge that they are helping create a product that affects millions
of people directly and even more people indirectly (because people use
Bugzilla as part of making other products). When somebody has
contributed a patch, WE NEED TO THANK THEM. It doesn't matter if the
patch is total crap and has to be re-written entirely, WE NEED TO THANK
THEM. They have put some work into this, and if we don't appreciate
that, THEY WILL LEAVE BEFORE THEY EVEN START. After all, most people get
little enough appreciation at their workplace--they stay there because
they get paid in money! They don't need to work FOR FREE with some other
organization if it also doesn't appreciate their work, or even worse,
assaults every aspect of their contribution before even thanking them
for it.
This isn't to say that we can't correct people on the faults in their
patches. "Kindness" does not include putting bad code into Bugzilla.
That isn't kind to anybody, including the contributor whose skills
probably need to improve, and who may go on believing that something
they did in error was in fact correct. We have to still be careful
reviewers and very good coders.
What this does mean is that in addition to telling people what's
*wrong* with their code, it's important to appreciate what's *right*
about their contribution, even if it's simply the fact that they took
the time to contribute. And you have to ACTUALLY TELL THE CONTRIBUTOR
THAT YOU APPRECIATE THE CONTRIBUTION. The more frequently and genuinely
that all of us do this, the more likely we are to retain the contributor.
Some people are hopeless programmers, and they're never going to get
better no matter how much you help them or how many reviews you do. For
these people, you don't have to imagine nice things to say about them--I
don't expect anybody to ever make up stuff that isn't true, just to be
nice. If somebody has demonstrated conclusively that they're never going
to get any better, or that they're just going to cause trouble one way
or another no matter what we do, the best thing you can do is simply
personally choose to ignore them, which is a personal decision that you
can make for yourself.
But that represents a very, very small percentage of total
contributors. For the vast majority of contributors, even though they're
a little rough around the edges for the first few months, the correct
action is to be helpful, kind, appreciative, and responsive.
3) A total absence of personal negativity. One thing that drives people
away from a project with lightning speed is when they get personally
attacked for attempting to do something positive. A "personal attack"
can be as little as an unpleasant joke about their code, instead of just
a straightforward technical description of what is wrong. Saying
something like, "What is wrong with you?" instead of actually leaving
some helpful comment. Disguising personal criticism as "an attempt to
help them code better" or "help them get along with others". (If
somebody does something that offends you, just tell them that it
offended you--that's usually totally enough to resolve the problem. You
don't have to pretend that other people have a problem just because you do.)
Look, I understand that sometimes coding and working on a collaborative
project with people who have different viewpoints can get REALLY
frustrating! I've been an offender in this area just as much as anybody
has been! But it's really not OK for us to insult other developers
personally just because we're frustrated with them personally.
If you're having a bad day, then please feel free to take a break.
Getting angry or upset with other developers in a personal way could
actually be more damaging to the long-term future of the Project than a
single day's worth of contributions would help.
And finally, if there's some contributor that you just can't live with,
please feel free to come to me, and I will absolutely do my best to work
it out, even if it's been a really long-term thing! I'm all about being
effective, too--I'm not going to just brush it off and say "well, maybe
it all will fix itself," I'll do something effective about it. And if
it's *me* that you have a problem with, please go to justdave about it,
and he'll talk to me about it--he's very good at that sort of thing. :-)
The most important thing is that you don't take it out on the person
you're unhappy with--I don't think that that helps anybody, and it
definitely harms the long-term future of the Project.
Now, #2 and #3 above pretty much cover my recommendations on how we
should behave toward new community members: be really, abnormally,
really, really kind, and don't be mean. But how do we address the
problem of the delay on reviews?
Well, here's my recommendations for how we handle the review problem:
1. I personally have started to prioritize review requests from
non-core contributors above anything else, including security work,
releases, ANYTHING. New contributors are my #1 priority, and I've
already seen this have a big effect on the number of contributions we're
getting from outside developers.
2. I think we should say that if you're a reviewer, it's your
responsibility to receive the review-requests emails, and we should
merge review-requests back into review@. You don't have to have them go
into your inbox, but you should at least receive and read them. I think
that this would get a lot of people more involved in review, because
everybody would get every review request, and even review from the
"wind" might start to work again. Particularly for the less-active
reviewers, this would give them more ability to pick and choose a few
items that they wanted to review now and then, and it would also let
each reviewer know if there was some area of their interest under review
so that they could chime in on it.
So whew! That's the results and my recommendations! :-) What do you
guys think? :-)
-Max
--
http://www.everythingsolved.com/
Competent, Friendly Bugzilla and Perl Services. Everything Else, too.
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=dev-apps...@lists.mozilla.org>
Max Kanat-Alexander
> * Freezes kill the community--every time we had a code freeze, the
> number of contributors would dramatically drop and then recover slowly.
Oh, and I forgot to address this! I think that we've actually handled
this recently, with our new freeze policy where we actually branch
immediately, instead of ever freezing, so I think that this won't be a
problem anymore in the future, although it's important that we pay equal
attention to trunk reviews and branch reviews, when we're in a
pre-release branch situation.
Justin Wood (Callek)
...
Allow me to just say THANK YOU to you and other contributors and core
people.
I have written a patch or two, at most to bugzilla, but have not done
more primarily because of time. (And dislike in general for perl).
Anyway, what you guys do is always welcome and thanked on my part, even
if I never say it. (There are still some pet peeves of mine, but they
almost all are mine alone due to some weird self-configs I did on BMO)
-
~Justin Wood (Callek)
Frédéric Buclin
> So my question was--what happened in 2005 that changed things so
> dramatically? After QUITE a bit of searching, I discovered the following:
I'm a bit surprised that nobody mentioned that 1) we have much less
visible bugs in our code since the QA team exists (July 2005:
https://wiki.mozilla.org/Bugzilla:QA), and so it's much less obvious for
people to fix bugs they don't see unless they look at the code itself.
Also, I heard here and there that 2) Bugzilla pretty much does all what
some old contributors needed and so don't need to contribute any more.
Why would you contribute for a feature you don't need? My personal
opinion is that 2005-2006 also reflects some maturity of the product. I
don't try to minimize what you mentioned in 1)-3), don't take me wrong.
> Now, #2 and #3 above pretty much cover my recommendations on how we
> should behave toward new community members: be really, abnormally,
> really, really kind, and don't be mean.
<troll>I hope this doesn't involve sending flowers to all our
contributors with a greeting card.</troll> More seriously, "abnormally,
really, really" sounds excessive to me and is not something I will fall
into.
> 1. I personally have started to prioritize review requests from
> non-core contributors above anything else, including security work,
Wow, security work should have the highest priority, IMO, because they
often block releases (same for blockers). And we from time to time delay
checkins because we are near a release date and we don't want to mess
the code too late in the game. So having releases done faster would help
focusing on new contributors once we have security bugs and blockers out
of our plate. At least, that's my point of view. To be clear, I'm since
early December ignoring *all* requests for review which aren't about
blockers, security bugs or targeted 3.6 or older, no matter who the
requester is or how cool the new feature is. My remaining free time and
motivation are devoted to QA. The sooner 3.6rc1 and 3.6 are released,
the sooner I will accept looking at other contributions again.
> 2. I think we should say that if you're a reviewer, it's your
> responsibility to receive the review-requests emails, and we should
> merge review-requests back into review@.
Please don't do that! reviewers@ and review-requests@ have different
goals. And I recently unsubscribed from review-requests@ for a reason
(after being 5 years in this mailing-list). Keep them separate, and just
email reviewers@ to subscribe to review-requests@ if they want to. I
don't see why you can/should/would be allowed to force them to read all
review requests. It's prone to be redirected to the trash bin directly.
And if you are the requestee, you already get review requests directly
anyway.
> everybody would get every review request, and even review from the
> "wind" might start to work again.
Out of the current 46 pending reviews in the Bugzilla product (4 of
which are more than 5 months old!), we have only 4 requests from the
wind. That's a pretty weak excuse to spam all reviewers with all review
requests.
Oh, and thanks for this research! ;)
LpSolit
Gervase Markham
Thanks for doing all this work ;-) I have a few questions and queries,
but that shouldn't undermine the fact that this is really useful data
gathering.
On 19/01/10 02:13, Max Kanat-Alexander wrote:
> * Freezes kill the community--every time we had a code freeze, the
> number of contributors would dramatically drop and then recover slowly.
Is this, at least partially, a statistical artifact of the fact that you
are measuring continuity as "contributes every month"? After all, a
freeze means fewer patches, so there may be more people who consider
themselves part of the community (and may have been attaching patches to
bugs to wait for when the tree reopens) but who did not actually check
in or provide a patch for the trunk because it was frozen.
Would it be worth instead doing some sort of three-month rolling average
(if you've contributed in month X, or X+1, or X-1, you get a tick for
month X)? Or perhaps look at attached patches in Bugzilla as well as
checkin data?
Did you find it was common for someone to leave the community and
return? If so, it might be an indication that your data isn't smoothed
enough in this way.
> * Despite this, the size of the community generally increased until the
> start of 2005, and then leveled off through 2005, and then started to
> slowly decrease after that. No matter how I reorganized the data, or how
> I analyzed it, 2005 was the critical year.
Is the raw data available to look at? What are the absolute numbers
here? Was the peak community size 4, or 40? Is the addition or removal
of a single contributor a significant factor in the numbers?
> 1) The "review" and "review-requests" mailing lists were split at the
> start of 2005. Before, all reviewers had at least seen every single
> review request. Now they had to subscribe to review-requests and
> actually *request* all that spam, which nobody was going to do. (I
> didn't even know about the review-requests list by the time I became a
> reviewer, which was after this point.) I think that this particularly
> hurt people's ability to get review "from the wind", since only old-time
> and very dedicated reviewers would have been on the list where those
> requests went.
I didn't even realise there was a review-requests mailing list. Or, if I
ever knew it, I've forgotten.
> I'd like to note here that nobody gave "the review process is too hard"
> as a reason for leaving. Some people here and there have banged the drum
> for a lowering of quality as an incentive to attract and retain
> community members, and there was absolutely no evidence in the survey
> that that would improve the situation at all.
As someone who might be interpreted as banging this drum, I think your
summary of the view (or, at least, my view :-) is not quite correct.
Firstly, I am not arguing for "lower quality". As you know, quality is
not a single number - "this patch has a quality of 45". And I have no
issues with someone being asked to change a patch at a code level or to
conform to style guidelines. But on different occasions, I have argued for:
- The acceptance of a patch which provides partial, useful,
functionality in pursuit of a larger goal, rather than waiting for the
submitter to implement the entire feature as envisaged by the team (but
not by the submitter)
- The acceptance of a patch which provides useful functionality, even if
the code would eventually be obsoleted by a rewrite of the feature which
is wanted but un-resourced and un-scheduled
- Within limits, the acceptance of a patch which solves a problem for an
existing user or set of users, even if the Bugzilla team don't
particularly like the way it's done, if the team has no plans to solve
that problem themselves 'the better way'. (The problem here is that
often the problem ends up lingering unsolved for years, to nobody's
benefit. This is what is generally known as "stop energy".)
Taking any patch is a trade-off between the plus of the problem solved
and the contributor encouraged, against the minus that the patch may not
be absolutely 100% as it would be if the reviewer had written it. We
need to consider whether we are weighing the second of those two factors
lightly enough.
As I understand it, your research was looking at people who became
community members and then stopped. My contention is that people who
have their patches rejected for reasons related to one of the above
never _become_ community members - and therefore would not have been
reached by your survey. (You talk more about initial acceptance below,
and that's great.) Therefore, I think your conclusion that the above
sorts of thing are not a factor may not actually be supportable by your
gathered evidence.
> Both of the former documentors who responded mentioned the lack of
> kudos from the community as one reason that they weren't motivated to
> stay in the community.
I'm a "former documenter", in that I used to work on the Bugzilla
documentation and now no longer do. My reasons for stopping were:
- The documentation build system was byzantine in its complexity, and
every time I got a new machine (or even when I didn't) it needed
setting up.
- The XML markup gave the ability to produce text, HTML and PDF, but it
made it very difficult to do things with graphics or layout in order
to make the documentation more readable. I had to remember an XML
vocabulary not used anywhere else in my online life.
- Documentation was treated like code, with patches having to be
prepared and reviewed, rather than like a wiki. Getting fixes in,
particularly rewrites and reorganizations, was just too much effort.
I've got a fairly thick skin, so the fact that a big documentation
reorganization was met with process-based obstructions rather than a
"wow, thanks for putting in all that work" didn't affect me too much :-)
> personal involvements. But nothing could be further from the truth--the
> personal interactions that people have with community members, the
> amount they feel appreciated, and the amount they feel assaulted, are
> actually the MOST IMPORTANT ASPECTS OF RETAINING COMMUNITY MEMBERS.
Have you read "The Art of Community" by Jono Bacon, recently published?
It's fantastic on this sort of thing - well worth a read. I read it
recently, cover to cover.
Also, if you are around at OSCON time, he runs the Community Leadership
Summit:
http://www.facebook.com/event.php?eid=186487039968
Anyone in the Bay Area at the time should consider attending.
> 1) Extremely rapid response to review requests. No matter how much
> review a contributor has to go through, they usually will go through it
> as long as there aren't LONG DELAYS between the request and the response.
Definitely. On the main Mozilla side, I've been trying for years to get
a system in place which tracks this across the project and nags the
appropriate people. The trick, in that case, is defining "appropriate
people".
> Some people are hopeless programmers, and they're never going to get
> better no matter how much you help them or how many reviews you do. For
> these people, you don't have to imagine nice things to say about them--I
> don't expect anybody to ever make up stuff that isn't true, just to be
> nice. If somebody has demonstrated conclusively that they're never going
> to get any better, or that they're just going to cause trouble one way
> or another no matter what we do, the best thing you can do is simply
> personally choose to ignore them, which is a personal decision that you
> can make for yourself.
I don't actually agree with that. I think the best thing you can do is
tell them why their contributions aren't being accepted, and say that
unfortunately you will not be able to make time in future to review
their contributions. This is possibly (although by no means certainly)
more likely to result in hurt feelings than just ignoring them, but it
is also much more likely that their life will improve as they go and do
something they are actually good at. You certainly don't have to be rude
or brusque when telling them, but I think you are doing them greater
good by telling them.
Also, bad coders might make great support people, or documenters (if
your documentation system isn't the functional and process equivalent of
writing code ;-).
> Look, I understand that sometimes coding and working on a collaborative
> project with people who have different viewpoints can get REALLY
> frustrating! I've been an offender in this area just as much as anybody
> has been! But it's really not OK for us to insult other developers
> personally just because we're frustrated with them personally.
I agree with all of what you write. Have you noticed any of this? This
isn't acceptable in a more broader Mozilla context, as well as in a
Bugzilla one, and I take it seriously when I find it.
> Well, here's my recommendations for how we handle the review problem:
>
> 1. I personally have started to prioritize review requests from
> non-core contributors above anything else, including security work,
> releases, ANYTHING. New contributors are my #1 priority, and I've
> already seen this have a big effect on the number of contributions we're
> getting from outside developers.
Fantastic! :-)
> 2. I think we should say that if you're a reviewer, it's your
> responsibility to receive the review-requests emails, and we should
> merge review-requests back into review@.
Sounds absolutely fine to me.
Gerv
Gervase Markham
> Wow, security work should have the highest priority, IMO, because they
> often block releases (same for blockers). And we from time to time delay
> checkins because we are near a release date and we don't want to mess
> the code too late in the game. So having releases done faster would help
> focusing on new contributors once we have security bugs and blockers out
> of our plate. At least, that's my point of view. To be clear, I'm since
> early December ignoring *all* requests for review which aren't about
> blockers, security bugs or targeted 3.6 or older, no matter who the
> requester is or how cool the new feature is. My remaining free time and
> motivation are devoted to QA. The sooner 3.6rc1 and 3.6 are released,
> the sooner I will accept looking at other contributions again.
I think that if Max is prioritizing patches from new contributors, then
not everyone else needs to. (If you are able to pass on reviews of such
patches that get sent to you, so much the better.) If he's doing new
people as the highest priority, and you are doing security, then that's
teamwork :-)
However, you can understand, I hope, that if _everyone_ adopted the
prioritization scheme you have, then it would be very hard for anyone
new to enter the community.
Gerv
Max Kanat-Alexander
inline below:
On 01/19/2010 04:38 AM, Gervase Markham wrote:
> Would it be worth instead doing some sort of three-month rolling average
> (if you've contributed in month X, or X+1, or X-1, you get a tick for
> month X)?
I think that actually wouldn't be statistically significant, because
the freeze harm was several months long, and the recovery from it was
slow over several months, for each freeze.
> Or perhaps look at attached patches in Bugzilla as well as
> checkin data?
Yeah, actually, I think that would be definitely worthwhile, at least
to get a sense of whether or not there was some problem with people
attaching patches but them never getting in, if nothing else.
> Did you find it was common for someone to leave the community and
> return?
I don't know, but them leaving the community at all is a problem, in
terms of what I was looking into. I mean, I want people to be
participating as much as they possibly can and want to.
> Is the raw data available to look at?
Yeah, I post it up somewhere, if you think that there's something to be
gained or some other investigation of value to be done with it.
> What are the absolute numbers
> here? Was the peak community size 4, or 40?
The peak was 25.
> Is the addition or removal
> of a single contributor a significant factor in the numbers?
No, not really. Also, there is so much data that it's not
month-to-month that matters, but the general trend of the graph, which
is pretty clear.
> I didn't even realise there was a review-requests mailing list. Or, if I
> ever knew it, I've forgotten.
Yeah, I didn't quite know about it either.
> - The acceptance of a patch which provides partial, useful,
> functionality in pursuit of a larger goal, rather than waiting for the
> submitter to implement the entire feature as envisaged by the team (but
> not by the submitter)
I used to accept things like this and I no longer do, because what
happens is that we actually end up releasing for years with a
half-implemented feature that causes more problems than it solves. I
spent almost the entire 3.6 development cycle "finishing" features that
should have been complete years ago (even ones that I wrote) and the 3.8
cycle is going to be taken up with a tremendous amount of that as well.
> - The acceptance of a patch which provides useful functionality, even if
> the code would eventually be obsoleted by a rewrite of the feature which
> is wanted but un-resourced and un-scheduled
That's something that can be accepted, *unless* the rewrite is
something that would actually only take a little more effort. For
example, component-watching is actually only a little bit easier than
everything-watching, and everything-watching might be more useful.
That's a fine line, and it's something that I know that I, at least,
could be pushed on, though, with a rational argument.
> - Within limits, the acceptance of a patch which solves a problem for an
> existing user or set of users, even if the Bugzilla team don't
> particularly like the way it's done, if the team has no plans to solve
> that problem themselves 'the better way'.
If "the Bugzilla team doesn't like the way it's done", that means that:
(a) The code is poorly architected
(b) The feature is implemented in a way that we think is not helpful
enough to justify the code complexity.
So in those cases, we won't accept patches, but we will provide review
comments and guidance on where to go with them. If the patch author
doesn't want to improve the patch, then we will probably eventually get
to it, but we won't accept a poorly-implemented feature, because that's
a disservice to our users.
> - The documentation build system was byzantine in its complexity, and
> every time I got a new machine (or even when I didn't) it needed
> setting up.
That is so true. We really need to switch to xmlto, but nobody's taken
the time to do that yet. I suspect that that will resolve this issue.
> - The XML markup gave the ability to produce text, HTML and PDF, but it
> made it very difficult to do things with graphics or layout in order
> to make the documentation more readable. I had to remember an XML
> vocabulary not used anywhere else in my online life.
Yeah, I'm not an enormous fan of that process, either.
> - Documentation was treated like code, with patches having to be
> prepared and reviewed, rather than like a wiki. Getting fixes in,
> particularly rewrites and reorganizations, was just too much effort.
Here's the thing about a Wiki--it requires a significant number of
editors who pay attention to it continuously and carefully. The
Win32Install instructions, which we have on the Wiki, became edited into
unusefulness while LpSolit and I weren't paying attention to them. Right
now we just don't have that many people that I think could effectively
function as technical editors of the documentation if it were on a Wiki.
Perhaps a trade-off would be keeping the installation instructions in
Docbook, and putting everything else into the Wiki. But then how would
we do localization? Would we still be able to ship documentation with
Bugzilla? (Is that even important?)
> Have you read "The Art of Community" by Jono Bacon, recently published?
> It's fantastic on this sort of thing - well worth a read. I read it
> recently, cover to cover.
No, I haven't. Do you think it's worth reading the whole book, or do
you think a Cliff's Notes version would help? :-)
>> 1) Extremely rapid response to review requests. No matter how much
>> review a contributor has to go through, they usually will go through it
>> as long as there aren't LONG DELAYS between the request and the response.
>
> Definitely. On the main Mozilla side, I've been trying for years to get
> a system in place which tracks this across the project and nags the
> appropriate people. The trick, in that case, is defining "appropriate
> people".
I could imagine a workable system here, I think. Something like, "After
X days, remind the requestee" and then if no response, "After Y days,
remind the module owner" and then if no response, "After Z days, remind
Somebody Important." Perhaps Somebody Important could be a Community
Manager for Mozilla, or something.
>> Some people are hopeless programmers, and they're never going to get
>> better no matter how much you help them or how many reviews you do. For
>> these people, you don't have to imagine nice things to say about them--I
>> don't expect anybody to ever make up stuff that isn't true, just to be
>> nice. If somebody has demonstrated conclusively that they're never going
>> to get any better, or that they're just going to cause trouble one way
>> or another no matter what we do, the best thing you can do is simply
>> personally choose to ignore them, which is a personal decision that you
>> can make for yourself.
>
> I don't actually agree with that. I think the best thing you can do is
> tell them why their contributions aren't being accepted, and say that
> unfortunately you will not be able to make time in future to review
> their contributions. This is possibly (although by no means certainly)
> more likely to result in hurt feelings than just ignoring them, but it
> is also much more likely that their life will improve as they go and do
> something they are actually good at. You certainly don't have to be rude
> or brusque when telling them, but I think you are doing them greater
> good by telling them.
Okay, you know what, I agree with you here, that at least as a first
attempt, it's more valuable to confront the individual, if you feel like
you can. I don't think that everybody has the requisite communication
skills (or wants to do this), so if a reviewer wants me or another
person to talk to them about it instead, then we'd be happy to. But if
nobody feels capable of talking to the individual, then another option
is to simply ignore the poor contributors who aren't getting better no
matter how many times they are reminded.
> Also, bad coders might make great support people, or documenters (if
> your documentation system isn't the functional and process equivalent of
> writing code ;-).
Yeah...I don't think I've ever seen a really poor coder write good
documentation or provide good support, though.
>> Look, I understand that sometimes coding and working on a collaborative
>> project with people who have different viewpoints can get REALLY
>> frustrating! I've been an offender in this area just as much as anybody
>> has been! But it's really not OK for us to insult other developers
>> personally just because we're frustrated with them personally.
>
> I agree with all of what you write. Have you noticed any of this? This
> isn't acceptable in a more broader Mozilla context, as well as in a
> Bugzilla one, and I take it seriously when I find it.
Well, I think that I notice it more in cases like, "That is a
stupid/ridiculous idea"--that's something that I consider a personal
attack on the individual instead of a constructive method of working
together. In general, I more often see personal insults *veiled* as
"professional disagreements" than I see actual personal insults.
Max Kanat-Alexander
> I'm a bit surprised that nobody mentioned that 1) we have much less
> visible bugs in our code since the QA team exists (July 2005:
> https://wiki.mozilla.org/Bugzilla:QA), and so it's much less obvious for
> people to fix bugs they don't see unless they look at the code itself.
That's possible. Some people do get involved by first providing a bug
fix. I think that people stay around to write features, though.
>> Now, #2 and #3 above pretty much cover my recommendations on how we
>> should behave toward new community members: be really, abnormally,
>> really, really kind, and don't be mean.
>
> <troll>I hope this doesn't involve sending flowers to all our
> contributors with a greeting card.</troll> More seriously, "abnormally,
> really, really" sounds excessive to me and is not something I will fall
> into.
Well, maybe not flowers. :-) I think that it's important to be nicer
than we otherwise might think that we have to be, though. Particularly
when we're all far away from each other and can't fix things through
direct in-person conversation, it's important not to cause upsets
between developers. Also, it's important to be really encouraging and
kind to people, so that they see that this is the type of community that
they want to be involved in. After all, nobody wants to voluntarily be
involved in a community that's abusive to them in any way, even if it's
professionally abusive. On the other hand, people do like to be a part
of a group that's kind of them and where they feel like their
contributions are important, so that's the kind of community I want to
create.
> Please don't do that! reviewers@ and review-requests@ have different
> goals. And I recently unsubscribed from review-requests@ for a reason
> (after being 5 years in this mailing-list). Keep them separate, and just
> email reviewers@ to subscribe to review-requests@ if they want to. I
> don't see why you can/should/would be allowed to force them to read all
> review requests.
The fact is, the project started to decline when we separated them, so
I think that we should try to rejoin them for at least a few months and
see if that changes things. This is something that we can *actually do*
to try to improve the size of the community. Do you want to be the only
person doing the majority of the development and reviews for the rest of
your life?
> And if you are the requestee, you already get review requests directly
> anyway.
Yes, but the point is to get the review requests a broader audience.
Maybe somebody requests review from you, but Gerv also feels like he
can/wants to say something about the patch, and he's able to get to it
before you. It allows all the reviewers to participate together as a
community.
Max Kanat-Alexander
>> Or perhaps look at attached patches in Bugzilla as well as
>> checkin data?
>
> to get a sense of whether or not there was some problem with people
> attaching patches but them never getting in, if nothing else.
Wow, so this analysis produces some very interesting data!
I see some definite trends:
1. From 2002-05 to 2004-01, the number of contributors decreases steadily.
2. From 2004-01 to about 2005-12, it increases significantly and regularly.
3. From 2006-01 to about 2007-12, it dives significantly and continuously.
4. From 2007-12 to 2008-05 it makes a comeback.
5. From 2008-05 to 2009-07 (which is where my data basically ends) it
stays stable, perhaps a slight rise.
These trends are actually very similar to the checkin trends, except
that they more clearly show a very specific rise to a peak at the *end*
of 2005, whereas the previous data looked roughly level for most of 2005.
I then did an analysis of the review flag, excluding review requests
and only checking denied and granted flags. The trends of how many
reviewers we have active at any given time correlate nearly-perfectly to
the curve of how many active contributors we have at any given time.
Also, interestingly, roughly 50% or more of all review flags set are
review+, and the percentage of r+ reviews granted in a month doesn't
correlate to the number of active contributors by any curve. I think it
doesn't matter whether we grant r+ or r-, as long as we grant it quickly
either way.
Here's the raw data for this analysis:
http://spreadsheets.google.com/ccc?key=0Ar05y8vgF7BhdDBlb3hJU1BKYWMySDFYdE1oOXlKTWc&hl=en
The next thing to check would be "average review delay" -- that is, the
average amount of time in a particular month that a patch attacher has
to wait for a review, including standard deviation. That would confirm
my hypothesis from above, though there were probably other factors
involved in how many contributors we had at any given time.
Gervase Markham
>> Did you find it was common for someone to leave the community and
>> return?
>
> I don't know, but them leaving the community at all is a problem, in
> terms of what I was looking into. I mean, I want people to be
> participating as much as they possibly can and want to.
What I meant was, if your definition of "leave" was "didn't contribute
for a month", then I'm sure I "left" and "rejoined" several times over
the last 10 years. I wanted to suggest that if you saw it was common for
people to "leave" and "rejoin", then perhaps you needed to average over
a longer time.
>> - The acceptance of a patch which provides partial, useful,
>> functionality in pursuit of a larger goal, rather than waiting for the
>> submitter to implement the entire feature as envisaged by the team (but
>> not by the submitter)
>
> I used to accept things like this and I no longer do, because what
> happens is that we actually end up releasing for years with a
> half-implemented feature that causes more problems than it solves. I
> spent almost the entire 3.6 development cycle "finishing" features that
> should have been complete years ago (even ones that I wrote) and the 3.8
> cycle is going to be taken up with a tremendous amount of that as well.
There's a difference between "half-implemented" and "half-implemented" :-)
If something is half-implemented in the sense that "it half-does A",
then I can see what you mean, and you are right to object. But if it's
half-implemented in the sense that "it does A, but it would be great if
it did B too, because really B and A go together, and in an ideal world
Bugzilla would do both A and B", then IMO we should still take it. It's
normally things in the latter category that I object to. For example,
Myk's component watching patch was rejected in favour of fixing a
"generic watching" bug. Five years later, we have neither.
>> - The acceptance of a patch which provides useful functionality, even if
>> the code would eventually be obsoleted by a rewrite of the feature which
>> is wanted but un-resourced and un-scheduled
>
> That's something that can be accepted, *unless* the rewrite is
> something that would actually only take a little more effort. For
> example, component-watching is actually only a little bit easier than
> everything-watching, and everything-watching might be more useful.
Great example - see above! You say it's only a little bit easier - but
five years ago we _had_ a patch for component watching, and we didn't
have one for everything-watching. The patch was turned down, and five
years later we still have neither feature.
I would politely suggest a better rule would be to ask the patch
submitter if they were interested in doing the generic feature. If they
said no, and if no-one else was working on it actively at that exact
moment, take the patch - even if you think generalising it isn't hard at
all. If it's not hard, then someone can do it as a further development :-)
>> - The documentation build system was byzantine in its complexity, and
>> every time I got a new machine (or even when I didn't) it needed
>> setting up.
>
> That is so true. We really need to switch to xmlto, but nobody's taken
> the time to do that yet. I suspect that that will resolve this issue.
I'm afraid I don't agree. The success of things like DevMo have been
their easy editability. Manual hacking of XML-based markup is not easy,
even if its compilation becomes easier.
> Here's the thing about a Wiki--it requires a significant number of
> editors who pay attention to it continuously and carefully. The
> Win32Install instructions, which we have on the Wiki, became edited into
> unusefulness while LpSolit and I weren't paying attention to them.
I presume you mean these edits?
https://wiki.mozilla.org/index.php?title=Bugzilla%3AWin32Install&diff=146397&oldid=123138
I think "edited into unusefulness" is an exaggeration.
Of course they were not as well-written as the rest of the document, but
they were clearly a record of someone's experiences doing the
installation and an attempt to make the document more useful. IMO,
reverting the lot without making any effort to see what was useful and
incorporating it was unfortunate.
And you have to set that against useful edits like these:
https://wiki.mozilla.org/index.php?title=Bugzilla%3AWin32Install&diff=174547&oldid=163572
> Right
> now we just don't have that many people that I think could effectively
> function as technical editors of the documentation if it were on a Wiki.
But what you have now is no-one _editing_ the documentation. You just
have people _changing_ it (and not many people doing that either). The
structure hasn't changed for years, and the limitations and complexities
of the authoring and build process mean there are no or few diagrams,
little useful formatting, few tables, and it's very hard to take the
overall view and say "Hmm, that belongs there, and that thing there, and
then we need to edit this bit, and this bit should be a separate page,
and..." because it's all too high-friction.
And when someone does an editing patch like that, it's really hard to
review, and so gets rejected ("come back with something smaller"). That
happened to me a few years ago :-(
Moving to a wiki is not going to be 100% improvement with no downside.
But I think we'd end up with much better, more accurate and more useful
documentation than we have now.
> Perhaps a trade-off would be keeping the installation instructions in
> Docbook, and putting everything else into the Wiki.
I think the installation instructions would benefit just as much, if not
more, from being in a wiki.
> But then how would
> we do localization?
According to http://www.bugzilla.org/docs/, we haven't had localized
documentation since 2.18. Regardless, we could have a parallel set of
wiki pages just as easily as we could have a parallel set of XML files.
> Would we still be able to ship documentation with
> Bugzilla? (Is that even important?)
I don't myself think it's particularly important, but I've never had to
install Bugzilla in a location where I couldn't access the Internet.
Even machines which are isolated from the net can often be installed by
ssh-ing in from another machine which does have access, or another
machine which can sit next to a laptop which does have access, on which
you can read the docs.
But I'm sure we could ship docs anyway. There's a MediaWiki2HTML Perl
module:
http://www.gerv.net/cgi-bin/mediawiki2html
>> Have you read "The Art of Community" by Jono Bacon, recently published?
>> It's fantastic on this sort of thing - well worth a read. I read it
>> recently, cover to cover.
>
> No, I haven't. Do you think it's worth reading the whole book, or do
> you think a Cliff's Notes version would help? :-)
I'd rather you read the book, because then you can hear it all from
someone other than me! ;-)
>> Definitely. On the main Mozilla side, I've been trying for years to get
>> a system in place which tracks this across the project and nags the
>> appropriate people. The trick, in that case, is defining "appropriate
>> people".
>
> I could imagine a workable system here, I think. Something like, "After
> X days, remind the requestee" and then if no response, "After Y days,
> remind the module owner" and then if no response, "After Z days, remind
> Somebody Important." Perhaps Somebody Important could be a Community
> Manager for Mozilla, or something.
The trick is mapping "I have a bug in product Foo and component Bar"
into "therefore the responsible module owner is Fred Bloggs".
https://spreadsheets.google.com/ccc?key=0AjgkRw6P9EKGcGNSLWhGaXI5eDBQbi1UeFYzajJaYmc&hl=en_GB
is our attempt thusfar.
>> Also, bad coders might make great support people, or documenters (if
>> your documentation system isn't the functional and process equivalent of
>> writing code ;-).
>
> Yeah...I don't think I've ever seen a really poor coder write good
> documentation or provide good support, though.
Lots of people who can't code provide good support. I'm sure various
admins in our newsgroups who help people have fairly rusty Perl skills.
And certainly the Firefox support community can't code. And a good
friend of mine was a mediocre coder, but a kick-ass QA person (although
he eventually went to theological college and is training to be an Old
Testament lecturer).
Gerv
Max Kanat-Alexander
> What I meant was, if your definition of "leave" was "didn't contribute
> for a month", then I'm sure I "left" and "rejoined" several times over
> the last 10 years. I wanted to suggest that if you saw it was common for
> people to "leave" and "rejoin", then perhaps you needed to average over
> a longer time.
Oh, if it's just a month-to-month issue, then that didn't really
matter, because I was looking at trends over several months (I think 5
at the fewest) instead of looking at the month-to-month relationship.
> There's a difference between "half-implemented" and "half-implemented" :-)
>
> If something is half-implemented in the sense that "it half-does A",
> then I can see what you mean, and you are right to object. But if it's
> half-implemented in the sense that "it does A, but it would be great if
> it did B too, because really B and A go together, and in an ideal world
> Bugzilla would do both A and B", then IMO we should still take it.
Yeah, I usually do take patches like that.
> It's
> normally things in the latter category that I object to. For example,
> Myk's component watching patch was rejected in favour of fixing a
> "generic watching" bug. Five years later, we have neither.
Yeah, and I would accept a component-watching thing that was based on a
generic-watching thing, now. In fact, that might even be the next patch
I write.
> I would politely suggest a better rule would be to ask the patch
> submitter if they were interested in doing the generic feature. If they
> said no, and if no-one else was working on it actively at that exact
> moment, take the patch - even if you think generalising it isn't hard at
> all. If it's not hard, then someone can do it as a further development :-)
That sounds like a reasonable suggestion that I can keep in mind in
similar situations.
>> Perhaps a trade-off would be keeping the installation instructions in
>> Docbook, and putting everything else into the Wiki.
>
> I think the installation instructions would benefit just as much, if not
> more, from being in a wiki.
Hmmm, well, at the moment I disagree. Perhaps an errata for the
installation or a FAQ about installation would do well in the Wiki though.
> According to http://www.bugzilla.org/docs/, we haven't had localized
> documentation since 2.18. Regardless, we could have a parallel set of
> wiki pages just as easily as we could have a parallel set of XML files.
Yeah, that's reasonable.
Actually, I think I'm more concerned about providing documentation for
separate versions, which we *do* do, and is still important, because
things really do change across versions. I don't currently know a simple
way to keep several different versions of the documentation in sync on a
Wiki in the way that we can do with patches and branches.
> I don't myself think it's particularly important, but I've never had to
> install Bugzilla in a location where I couldn't access the Internet.
> Even machines which are isolated from the net can often be installed by
> ssh-ing in from another machine which does have access, or another
> machine which can sit next to a laptop which does have access, on which
> you can read the docs.
>
> But I'm sure we could ship docs anyway. There's a MediaWiki2HTML Perl
> module:
> http://www.gerv.net/cgi-bin/mediawiki2html
Yeah, that seems reasonable. I don't think it's that important either.
> The trick is mapping "I have a bug in product Foo and component Bar"
> into "therefore the responsible module owner is Fred Bloggs".
> https://spreadsheets.google.com/ccc?key=0AjgkRw6P9EKGcGNSLWhGaXI5eDBQbi1UeFYzajJaYmc&hl=en_GB
> is our attempt thusfar.
Yeah, maybe a Bugzilla Extension that added a "manager" or something to
a Component would help keep track of it a bit better.
>>> Also, bad coders might make great support people, or documenters (if
>>> your documentation system isn't the functional and process equivalent of
>>> writing code ;-).
>>
>> Yeah...I don't think I've ever seen a really poor coder write good
>> documentation or provide good support, though.
>
> Lots of people who can't code provide good support.
No, that's true for sure. I've just never met somebody who *started* as
a coder, and was bad, and then went on to be a good support technician.
Frédéric Buclin
We have french documentation since 3.0:
http://www.frenchmozilla.fr/bugzilla/
LpSolit