|Contributor pathways, engagement points and bug mentoring||Mike Hoye||3/30/14 1:49 PM|
Hi, everyone -
Like the title says, I've written up a couple of things about:
Code-specific contributor pathways:
Engagement points: https://wiki.mozilla.org/Contribute/Coding/Engagement
and Mentoring / Good First Bugs:
... at some length, because I love me some big words and that's how I
roll. I'll spare you the details; they're there if you're interested. In
- The Pathways doc outlines contribution tiers, how we're going to
measure engagement at every level, and some starting practices for
retention and reward. Most of it is generally of the form "We know it's
important to get back to new contributors promptly, so please let's do
that. When they finish a bug, thank them and suggest the next one."
- The Engagement doc lists the various ways that various people can put
interesting stuff in front of interested peoples' eyes. If I had to fit
this doc into a tweet, it would be "Choice paralysis is real, you don't
need to reinvent any wheels, we're here to help."
- The Mentoring doc spells out what makes a good first bug and a
well-functioning mentoring relationship. At the moment, most of our
good-first-bugs just aren't and a lot of contributors are getting
unintentionally pitched overboard -
https://bugzilla.mozilla.org/show_bug.cgi?id=989476 aims to fix one
such example - so I intend make auditing and automating those processes
a q2 goal.
I'd like to get us to contributor parity - the same number of paid
engineers as volunteer contributors getting patches in every product
cycle - within the next four releases. So I'm going to try to make that
|Re: Contributor pathways, engagement points and bug mentoring||Nick Alexander||3/30/14 5:31 PM|
On 2014-03-30, 1:49 PM, Mike Hoye wrote:Just wanted to say that these are valuable (I specifically bookmarked
Pathways) and that they led me to all the curated content under
Contribute/. Thanks for spending time to write down concrete best
practices for how to help grow our Mozilla community.
|Re: Contributor pathways, engagement points and bug mentoring||Josh Matthews||3/30/14 7:03 PM|
On 03/30/2014 04:49 PM, Mike Hoye wrote:One strategy I've started suggesting to mentors is to _offer_ to suggest
a next bug to work on. This allows the mentor to acknowledge the success
immediately, while giving them time to prepare an appropriate followup
|Re: Contributor pathways, engagement points and bug mentoring||Chris Peterson||3/30/14 10:15 PM|
On 3/30/14, 1:49 PM, Mike Hoye wrote:This is an exciting plan! What is our current ratio of patches from
employee vs volunteer contributors? How do you determine whether a patch
is from Mozilla employee? Many employees' Bugzilla accounts and patches
use their personal email addresses instead of mozilla.com addresses.
|Re: Contributor pathways, engagement points and bug mentoring||Josh Matthews||3/30/14 10:36 PM|
Employee status is derived from the list of email addresses obtained via
https://github.com/jdm/outrider/blob/master/generate_emails.py , which
scrapes from all email fields in the internal phonebook. I am aware of
very few inaccuracies, since most employees have been happy to update
their entry with addresses they use.
|Re: Contributor pathways, engagement points and bug mentoring||Mike Hoye||3/31/14 8:25 AM|
On 2014-03-31, 1:15 AM, Chris Peterson wrote:We're generally floating around 50% per rapid release now; I'll have
some better historical and trend data to share shortly.
|Re: Contributor pathways, engagement points and bug mentoring||David Rajchenbach-Teller||4/3/14 4:09 AM|
At some point, we should start thinking about a contributor dashboard,
which would handle both Bugs Ahoy and "Find a next bug".
> dev-planning mailing list
David Rajchenbach-Teller, PhD
Performance Team, Mozilla
|Re: Contributor pathways, engagement points and bug mentoring||Gavin Sharp||4/6/14 12:32 PM|
Great stuff. A couple of points of feedback:
This should probably include firefox-dev?
I think this is missing some important bug triage activities. Beyond
"filing" and "reproducing" and "creating a testcase", there's a lot of
value in "marking duplicates" or "confirming bugs" (and the
prerequisite step to those, "gaining editbugs privileges"). These are
how I got sucked in to Mozilla, and it's what eventually led me to
> Code-specific contributor pathways:
> - The Pathways doc outlines contribution tiers, how we're going to measure
> - The Engagement doc lists the various ways that various people can put
> I'd like to get us to contributor parity - the same number of paid engineers> Thanks.
> - mhoye
|Re: Contributor pathways, engagement points and bug mentoring||Mike Hoye||4/7/14 7:04 AM|
On 2014-04-06, 3:32 PM, Gavin Sharp wrote:It should - there are a few missing fora that need to go in there,
>A strong point, I'll add that.
|Re: Contributor pathways, engagement points and bug mentoring||cb...@mozilla.com||4/15/14 4:41 AM|
I think the part about
" That a contributor is recognized for their efforts, even if this is a simple as thanks, and "
is super important.
I do now as one of the sheriffs some of the checkin-needed requests and it happens sometimes that i see a bug from a new contributor who submits his/her first patch that got reviewed and now landed in our codebase.
Doing a comment in bugzilla like "thanks for contributing and welcome to mozilla" is i guess a important thing to let them feel welcome and that their work is recognized!
So i think that (while its simple) a important thing to keep people engaged into mozilla.
|Re: Contributor pathways, engagement points and bug mentoring||Mike Hoye||4/15/14 6:54 AM|
Ah, yeah - to clear this up, this really means "we have half as many
non-employees contributing as employees per rapid-release". And that's
people - as you might imagine, the ratio in terms of patches contributed
is much lower.
Some good news, following on from this last two weeks, is that uptake on
@startmozilla seems to be trending in the right direction and new people
are still showing up to work on stuff. "New contributors" and
"contributor retention" are only proxy measurements for the high-value,
long-term contributors we really want, obviously, but as metrics go
they're non-terrible and and it looks like we can move the needles on
those gauges meaningfully in the near term.
Some bad news is that the cultural impedance between our reliance on
NeedInfo and new contributors arriving from "ask" cultures is worse than
To sum up - by relying exclusively on needinfo to know what bugs to look
at, we're losing a lot of potential contributors who ask for permission
to participate in a bug comment, get a demoralizing silence in response
and leave. The forty or so instances of this I've found - and these are
just the ones I've found, mind you - all have that same pattern, and
possibly worse, they're almost all coming from markets - mostly India
and southeast Asia - where we really want to make a lot of new friends.
I've filed bug 989476 -
https://bugzilla.mozilla.org/show_bug.cgi?id=989476 - to mitigate
somewhat, the TL;DR being is that if you've agreed to be a mentor in a
bug, any comment from a "new to bugzilla" contributor will trip the
needinfo flag. If you've got strong feelings about that for or against,
that's where to register them.
|Re: Contributor pathways, engagement points and bug mentoring||Gavin Sharp||4/15/14 2:29 PM|
On Tue, Apr 15, 2014 at 6:54 AM, Mike Hoye <mh...@mozilla.com> wrote:Speaking based on personal experience, I think such comments tend to
be ignored because they very infrequently lead to a positive new
contributor interaction, and not because of any "reliance on
Feels like there's probably room to tackle this from the other angle -
i.e. teaching people they don't need to "ask first".
|Re: Contributor pathways, engagement points and bug mentoring||Gijs Kruitbosch||4/15/14 3:09 PM|
On 15/04/2014 22:29, Gavin Sharp wrote:This.
And also, hard choices about what kind of engagement we do and how
selective we are about contributors that don't seem to "get" this or be
able to work without asking. And more asking. And more asking.
I've had a few experiences now where people responding to good first
bugs or mentored bugs had to be handheld to such a degree (ie,
essentially needed literal instructions along the line of "what exact,
character-by-character, code changes do I need to make to fix this
bug"), and even didn't manage to do what was asked ("please run these
tests with ./mach mochitest-browser foo" - "I've run them and they pass"
- push code, break tree because those exact tests fail).
This makes me reluctant to sign up to mentoring bugs, because this way,
bugs repeatedly take orders of magnitude more time from me than they
would have done if I'd fixed them myself, with no discernible advantage
to myself or the project (aforementioned types of contributors move on
to other bugs with the same method, taking just as much time from myself
or others, and/or disappear).
To be clear, I understand that we need to invest in enlarging the
community, and that this takes time and effort, and that it's expected
and OK that the first (few) bug(s) that someone fixes take more time
from the mentor than fixing it would have done had they done it
themselves, and that not everyone sticks around. And that's fine. But
the degree to which I've seen people's (not just my own) time being
eaten up by certain aspiring contributors was very demotivating, and I
wonder what, if anything, we can/are do(ing) to fix this.
|Re: Contributor pathways, engagement points and bug mentoring||Ehsan Akhgari||4/15/14 3:17 PM|
On 2014-04-15, 6:09 PM, Gijs Kruitbosch wrote:That is very unfair to people who are coming from cultures where jumping
in the middle of something without asking for permission first is
considered a very impolite thing to do. We're not talking about typical
Western cultures that we mostly operate under.
I think this is a separate issue. Not every contributor is going to be
as savvy as all others.
Speaking from my personal experience when I first started to contribute
to Mozilla, it took me a *month* to be able to build the code base
locally. During that time a couple of individuals continued to try
their best to help me along the way. And without that, I would have
never stuck around the project.
The goal of attracting new contributors is not to help our engineers get
stuff done faster. It is to attract new talent to the project who can
help drive it forward in ways that no single contributor can. It's OK
if someone doesn't have time to help with that (I rarely have enough
time for that these days myself) but let's not forget the bigger picture
|RE: Contributor pathways, engagement points and bug mentoring||Mike Connor||4/15/14 3:36 PM|
To add to this: a great many of us "didn't get it" early on. Mozilla code
is hard and daunting at first, and there are many ways to get lost. If I
didn't have bz/dwitte/timeless patiently answering questions, I'd have
bounced in 2003 without ever getting a build working, let alone
accomplishing anything I've done in the ensuing decade. Without outing
people on this list, there are many established Firefox hackers who needed
a lot of support in the early days. If we aren't providing that support
and guidance, we'll miss out.
Is it fun? Not always. Do we all have other things we could be doing?
Sure. Is it an essential part of how we bring people in (and how we learn
to be better ourselves)? Absolutely.
Behalf Of Ehsan Akhgari
Sent: April 15, 2014 6:17 PM
To: Gijs Kruitbosch; dev-pl...@lists.mozilla.org
Subject: Re: Contributor pathways, engagement points and bug mentoring
[Mike Connor] <snip>
|Re: Contributor pathways, engagement points and bug mentoring||Gijs Kruitbosch||4/15/14 3:58 PM|
It seems my message wasn't clear, although I sort of saw this response
coming and tried to pre-empt it - evidently I failed.
I'm not complaining about people generally needing, and asking for, help.
9-odd years ago, when I started to contribute as a volunteer, I also
needed a lot of help. Heck, to "out" myself here: I needed help
understanding some of the programming language at issue (nobody had ever
told me String.replace() could take a function as the second argument!
(and JS was different from VBScript (yes...) in surprising ways)).
If/when I fix the occasional C++/objc bug, I need help with some of our
internals (and sometimes the language - clang helped a lot because the
error messages are so much better), still.
But needing help and needing literally all the things handed to you on a
silver platter are not the same.
I actually think we should be spending more time than we are on engaging
with new contributors (and it's something I plan on doing now that
Australis is finally being released). However, I also think we should be
careful about where and how we spend our efforts.
The time all of us, collectively, can spend on helping new contributors
up to speed is a limited resource. It would be nice if we could optimize
its usage better than we currently do.
Perhaps I (and one or two other people who mentored similar contributors
and with whom I've discussed this when these particular contributors
were active) were the only people affected, and the issue I'm bringing
up is such a minority case we don't need to worry about it. I hope so!
But if not, we should figure out how/where to draw a line.
It's never felt right to me to say "no" to someone who wants to own a
bug, but if previous evidence strongly suggests they don't stand a
chance of fixing them and will "occupy" the mentored bug when we only
have a limited number of them (seriously,
http://www.joshmatthews.net/bugsahoy/?ff=1&unowned=1 has very few bugs
that have never been (unsuccessfully) owned, so bugs with the right
difficulty level for first contributors are clearly hard to come by**),
then I would love to have a better way to respond than letting the
inevitable happen when I do assign them the bug.
** which probably means we should also think harder on the fx desktop
team about which bugs could be such bugs, and how to make more of them!
Many of those bugs are devtools bugs, and there doesn't appear to be a
way to filter those as compared to the rest of Fx desktop; Without data,
I do believe the devtools team has been doing a better job in terms of
mentored bugs than we have.
|Re: Contributor pathways, engagement points and bug mentoring||Gavin Sharp||4/15/14 3:59 PM|
Straw man alert: No one is arguing for "not providing support and guidance".
Nor did I intend to suggest that "teaching people not to ask" is the
only option - I very explicitly mentioned that this should be tackled
from both angles.
My point (and I think Gijs' as well) is just that not all people who
express a desire to contribute can be successful in doing so, for
various reasons, and "respond more to people's comments" on its own is
unlikely to be a cost-effective or scalable solution. We need to have
systematic solutions for avoiding expending large amounts of effort
for minimal return, and efficient ways to direct potential
contributors to the right opportunities.
I know mhoye has given those problems lots of thought, so this isn't
news to him, but it felt worth clarifying in the context of this
|Re: Contributor pathways, engagement points and bug mentoring||Trevor Saunders||4/15/14 4:08 PM|
On Tue, Apr 15, 2014 at 11:58:32PM +0100, Gijs Kruitbosch wrote:yes, this
I've certainly seen this type of person too.
yes please. Also if we do better at sending contributors who have no
hope of fixing the bug someplace else I hope we'll spend more time
helping people who are or at least will be useful in the future, and be
less frustrated while we do it.
|Re: Contributor pathways, engagement points and bug mentoring||Ehsan Akhgari||4/15/14 4:30 PM|
It seems we're mostly in violent agreement!
Yeah agreed. I think it's totally OK to suggest different things for
those types of contributors to spend their time on if you believe their
contribution won't get anywhere based on past experience. We have quite
a bit of non-technical ways of contributing!
|Re: Contributor pathways, engagement points and bug mentoring||Mike Hoye||4/15/14 7:18 PM|
On 2014-04-15, 6:59 PM, Gavin Sharp wrote:There's a lot to unpack here.
There's no question that there will be some people who will not be
effective contributors after any amount of effort by any number of
established Mozillians. It's unfortunate, but those people exist and
nobody who takes on a mentoring role should feel bad or hesitate for a
second to say somebody isn't ready, or needs to show some work or
acquire some more basic skills before they can proceed. Mentoring isn't
doing the work for someone, and if that's what it's turning into,
absolutely try to direct that contributor to somewhere they might be
more successful. We have some great introductory and non-code
contribution channels now - #introduction, SuMo, QA, Localization, the
Reps, a bunch of others; the systematic solution I'm hoping we can adopt
in the near term is "know the other engagement pathways and send people
to them as soon as it occurs to you that you should."
Nevertheless, I don't think that the possibility that somebody _might_
be a time-sink means we can leave them hanging,
I'm making a very narrow claim, here. All the data I've got says that
these are not people who are expressing a desire to contribute and then
failing to deliver; these are people who are expressing a desire to
contribute and then inadvertently being ignored. I know the solution
I've proposed isn't all that pretty, but I'm looking at a set of
mismatched expectations between how people interact with a service we
own and control outright and how people interact with other people in
about three-fifths of the entire world. In that context an inelegant
solution we can push to production over the weekend looks like a pretty
I agree that responding to more people's comments on its own isn't
cost-effective or scaleable, but getting new contributors to the point
where they can mentor the next generation of contributors themselves,
that scales like crazy. In fact, until late in the industrial
revolution, that was _the only thing that had ever_ scaled. And I don't
see a way to get from here to a million Mozillians that doesn't involve
both talking to lots of new people and leveling up the people we have.
If we get to the point that just talking to the number of new
contributors who want to fix bugs becomes an unsupportable burden for
Mozilla's engineers we will absolutely have to figure out how to address
that, but from up here in the cheap seats I bet that looks an awful lot
|Re: Contributor pathways, engagement points and bug mentoring||Ehsan Akhgari||4/16/14 8:53 AM|
On Tue, Apr 15, 2014 at 10:18 PM, Mike Hoye <mh...@mozilla.com> wrote:To get back to the original purpose of this thread, I actually question
Gavin's assertion about the cost effectiveness and scalability of answering
"yes please, you can work on this bug". As someone who has worked in two
very different cultures, I think it's very easy to dismiss these types of
asks before working on a bug as an indicator that the requestor is not
serious etc but that is far from the truth, they are just trying to be
polite within the cultural norms that they are used to.
That is a great problem to have, and we are *far* from it.
|Re: Contributor pathways, engagement points and bug mentoring||Robert Kaiser||4/16/14 9:31 AM|
Gijs Kruitbosch schrieb:
> ** which probably means we should also think harder on the fx desktopThat's surely a good idea!
|Re: Contributor pathways, engagement points and bug mentoring||Benjamin Smedberg||4/16/14 12:29 PM|
On 4/16/2014 3:23 PM, Josh Matthews wrote:
> There is also nothing wrong with unassigning someone who stops
> responding (a needinfo with a week's leniency seems polite to me).
The recommendation at
https://wiki.mozilla.org/Contribute/Coding/Mentoring is that we don't
assign bugs to new contributors until they've produced the patch. So the
response to "I'd like to work on this bug, please assign it to me"
should for new contributors be "Please feel free to work on this bug. We
don't assign bugs to new contributors until they have created a patch,
but please feel free to ask for help here as bug comments."
|Re: Contributor pathways, engagement points and bug mentoring||Josh Matthews||4/16/14 12:23 PM|
On 04/16/2014 02:58 PM, Chris Peterson wrote:
> On 4/16/14, 11:01 AM, Gavin Sharp wrote:
>> But, my personal experience has shown that a high percentage of the
>> people who ask "can you assign this bug to me" or some variant of it,
>> aren't actually ready to have the bug assigned to them. I have
>> assigned bugs, and spent time providing helpful comments, only to get
>> no response or to discover that the person is so far from being able
>> to contribute (for various reasons) that it's not an effective use of
>> my time to help them with that bug. Those experiences are what led to
>> me "ignore" such requests, for better or worse, and the needinfo
>> proposal isn't going to address that problem.
> Maybe we can craft some polite canned responses to new contributor's bug
> questions. If they have not met some minimum requirements like "I have
> successfully built Firefox and created a test patch", then we can direct
> them from Bugzilla to #introduction and the wiki.
Exactly. This is the role of #introduction; mentors should not feel
responsible for solving build environment issues that are not a direct
result of changes made in order to solve the bug under discussion. Off
the top of my head:
"Have you compiled Firefox from source? If not, please see
https://developer.mozilla.org/en/Simple_Firefox_build and direct further
questions to #introduction on http://irc.mozilla.org. When you a build
environment prepared, please feel free to start work on this bug
immediately, and I will try to answer any specific questions you have
about this task."
|Re: Contributor pathways, engagement points and bug mentoring||Gavin Sharp||4/16/14 11:01 AM|
On Wed, Apr 16, 2014 at 8:53 AM, Ehsan Akhgari <ehsan....@gmail.com> wrote:I think you misunderstood my claim.
If all someone needs is a "yes please, go right ahead! Here are some
tips:", we should definitely provide it to them. If adding needinfos
helps with that, that's great, and I don't object to it.
So we need to tackle both sides of the problem. That's all I'm saying.
|Re: Contributor pathways, engagement points and bug mentoring||Chris Peterson||4/16/14 11:58 AM|
On 4/16/14, 11:01 AM, Gavin Sharp wrote:
> But, my personal experience has shown that a high percentage of the
> people who ask "can you assign this bug to me" or some variant of it,
> aren't actually ready to have the bug assigned to them. I have
> assigned bugs, and spent time providing helpful comments, only to get
> no response or to discover that the person is so far from being able
> to contribute (for various reasons) that it's not an effective use of
> my time to help them with that bug. Those experiences are what led to
> me "ignore" such requests, for better or worse, and the needinfo
> proposal isn't going to address that problem.
Maybe we can craft some polite canned responses to new contributor's bug
questions. If they have not met some minimum requirements like "I have
successfully built Firefox and created a test patch", then we can direct
them from Bugzilla to #introduction and the wiki.
|Re: Contributor pathways, engagement points and bug mentoring||Michael Connor||4/21/14 7:24 AM|
On Apr 15, 2014, at 6:59 PM, Gavin Sharp <ga...@gavinsharp.com> wrote:
> We need to have
> systematic solutions for avoiding expending large amounts of effort
> for minimal return, and efficient ways to direct potential
> contributors to the right opportunities.
I think this gets held up a lot, but usually in the context of people arguing that we should make calls quicker on people to avoid sinking too much effort into an individual that won’t deliver a good enough return on that investment. My point is that we often can’t know, without investing a significant amount of time, whether someone will be an important contributor over the long run.
Largely, I think we worry a lot about the false positive case (when we invest too much in someone who wasn’t worth it), and don’t spend as much time thinking about the false negative case (when we fail to invest enough time in someone who would be an important contributor). Perhaps more importantly, I think we value the “lost” time for the false positive far more than we value the contributions we’re losing from the false negative case. That’s natural, but I think we can and should do better.
|How to best engage with and invest time in new/beginning contributors (was: Re: Contributor pathways, engagement points and bug mentoring)||Gijs Kruitbosch||4/21/14 7:41 AM|
On 21/04/2014 15:24, Michael Connor wrote:I didn't mean to hold this up in terms of "we need quicker calls" but
more in terms of "how do we communicate such a call". I find it hard to
say "no" generally, and even harder in these circumstances.
However, I think that the below point is more important:
Two points here:
1) as I pointed out, the false positive and false negative cases are
linked in that supplies are (relatively) low, and more than a few of the
bugs in http://www.joshmatthews.net/bugsahoy/?ff=1&unowned=1 were at one
point owned and went unfixed. All the time in which bugs are assigned to
people who don't fix them is time they spend making it harder for the
"right" people to find bugs to work on.
2) How can we reduce the number of false negatives? This is intended as
a genuine question - I would love to be able to help in that department.
Could we try to e.g. get team members to identify five mentored bugs a
week or something? Send swag to people for fixing mentored bugs? Invite
them for work weeks? Organize "fix mentored bugs" competitions/days?
Make it easier for people to fix bugs by improving the "effort curve"
for contributing? Something else? Do we (does Mike) have data on some of
I'm sure Mike Hoye has lots of thoughts on this subject, but it's a
while away from his original post, so I'm threadsplitting (at least
subject-wise, ymmv depending on how you're interacting with this
|Re: How to best engage with and invest time in new/beginning contributors (was: Re: Contributor pathways, engagement points and bug mentoring)||Benjamin Kerensa||4/21/14 9:27 AM|
On Mon, Apr 21, 2014 at 7:41 AM, Gijs KruitboschIs there any current policies in place in how you work with
contributors and at what point you stop investing time into a
contributor or group of contributors? Do you have strict criteria you
look for currently such as code examples (Github, Launchpad etc)
before investing time in a contributor?
I don't think this is something that can be easily solved and I think
all open source projects struggle with investing time in contributors
who just do not work out. On the other hand some other open source
projects do have employees who role is strictly connecting good
contributors and filtering out ones that are not going to work out.
Do we have any tools to check bugs that are assigned for periods of
time and perhaps leave a canned message and unassign the contributor?
I think all of those forms of recognition are good ways to recognize
contributors but I hope that some policy surrounding when each is
offered is developed. For instance I think in the case of inviting
someone to a work week they should be a core contributor to your team
that is contributing at least part time what a paid contributor might.
Also sometimes an e-mail or note is much more rewarding to
contributors then a t-shirt or swag. Just knowing that they are
helping and providing some value is pretty powerful recognition too.
|Re: Contributor pathways, engagement points and bug mentoring||Gavin Sharp||4/21/14 12:01 PM|
I worry about the "false positive" case not just because it represents
"wasted effort", but because it's a demotivating experience for the mentor,
which can build on itself and make people less likely to take a chance and
invest in a new contributor in the future. That leads directly to the
"false negative case" that you're describing.
These aren't independent problems - all I'm saying is that we shouldn't
treat them as such, and we need to address them both.
> I think this gets held up a lot, but usually in the context of people
> Largely, I think we worry a lot about the false positive case (when we> — Mike
|Re: Contributor pathways, engagement points and bug mentoring||Michael Connor||4/21/14 12:22 PM|
This is an interesting perspective. I’ve mentored plenty of people who didn’t succeed or follow through, and I don’t think I found it demotivating or made me less willing to help others, but that could be an expectations issue. I think I always recognized that I was accepting the false positives to make sure I caught as many good hackers as I could find. It probably helped that my initial job description had community building as 50% of my time, but it was how I was working even when I was still a volunteer.
Describing it as “taking a chance” implies there’s a risk involved for the mentor, but I’m not sure what’s being risked. Is it reputation? Time? Self-confidence (in terms of self-belief in mentoring skills)? Can we help people understand that failures are an expected part of the process, and a necessary evil?
I completely agree that they’re not independent problems. I think it’s a question of “which direction are we happier failing in?” For me, it’s false positives, because I believe that the cost of false positives will be lower than the benefit we’ll get from fewer false negatives. It’d be good to figure out how we measure that to be sure that math holds.
|Re: Contributor pathways, engagement points and bug mentoring||Gijs Kruitbosch||4/21/14 12:42 PM|
On 21/04/2014 20:22, Michael Connor wrote:Can you give some bugs as examples? How recent was this? I didn't find
anything with the bugsahoy whiteboard tags in bugzilla (for "mconnor")
so I'm wondering if we're just seeing very different types of
contributors now than we have in the past, and so perhaps it's an
experiences rather than expectations issue. :-)
|Re: Contributor pathways, engagement points and bug mentoring||Mike Hoye||4/21/14 12:54 PM|
On 2014-04-21, 3:22 PM, Michael Connor wrote:For what it's worth, all the developers I've interviewed last year spoke
about the failure of the relationship - not "failure to complete a bug",
mind you, but specifically the failure of the relationship - as being a
major demotivator. So while it's not universally the case, the
demoralizing-false-positive is certainly a real thing. The way I'm
trying to minimize the downside there is by asking people to get into
these relationships cautiously, and to be willing to size up
contributors and redirect enthusiastic but out-of-their-depth people to
something they'll have a good shot at.
We don't have a lot of support for the practice of mentoring in place
right now, which makes a failed mentoring relationship look (and
certainly feel) like a personal failure on the part of the engineer,
rather than the systemic problem that IMO it really represents; we're on
our way there, though, between the mentoring fields coming into Bugzilla
and some better documentation, hopefully this will make for more
functional and easy-to-manage contributor relationships.
|Re: How to best engage with and invest time in new/beginning contributors||Mike Hoye||4/21/14 1:01 PM|
On 2014-04-21, 10:41 AM, Gijs Kruitbosch wrote:
>> I think this gets held up a lot, but usually in the context of peopleSo, I have more to say about this - when I'm back from my trip tomorrow,
one hopes! - but I think that:
- it's pretty clear when somebody is thrashing, rather than making
forward progress, and
- the real challenge is actually making "the call", rather than knowing
that you should.
With regards to what "the call" means, my theory is that having a bunch
of different options for how and where an engineer can send a strugging
contributor close to hand will make this much easier and happier
experience, so I'm working on building those out right now.
|Re: Contributor pathways, engagement points and bug mentoring||Gavin Sharp||4/21/14 1:02 PM|
You can't see how mentors might be reluctant to invest time in helping
contributors if a high percentage of the their past efforts have not been
successful in any form? This seems fairly obvious to me. No one is
expecting a 100% "success" rate - but there's no getting around the fact
that a 100% (or 90%) "failure" rate is going to have an impact on your
I fully support encouraging people to invest more in mentoring, and it
makes sense to bias our cost/benefit analysis to the longer term rather
than focusing on short term returns. But you can't boil this down to just
"pick one direction to fail in", because failing too much in the "false
positive" case makes you fail more in the "false negative" case too.
> issue. I think I always recognized that I was accepting the false
> On Apr 21, 2014, at 3:01 PM, Gavin Sharp <ga...@gavinsharp.com> wrote:
> > I think this gets held up a lot, but usually in the context of people
|Re: How to best engage with and invest time in new/beginning contributors||Mike Hoye||4/21/14 1:13 PM|
On 2014-04-21, 12:27 PM, Benjamin Kerensa wrote:I wrote a bunch of stuff up here about the mentor's role and expectations:
They're intended to be helpful guidelines, not policies. No policy or
criteria is going to be able to replace or do better than smart people
who care exercising their best judgement.
Yeah, you can't "solve" people.
Keeping an eye on this situation is my responsibility; I'm hoping to
automate some of it, but either way.
I've outlined a bit of that here:
https://wiki.mozilla.org/Contribute/Coding/Pathways - it's a work in
progress, but it speaks to some of your points.
...but the situation is actually worse than you describe - after a
certain point sending a high-value contributor a t-shirt or some
stickers is kind of insulting; at that point, the rewards that matter
are involvement, recognition and greater responsibility.
|Re: How to best engage with and invest time in new/beginning contributors (was: Re: Contributor pathways, engagement points and bug mentoring)||Michael Connor||4/21/14 3:42 PM|
These are very difficult conversations to have, for sure. I think a big part of the art of mentoring is about helping people find the right path for them. It’s not just “how do I do this thing?” but “how do I succeed?” If they’re on the wrong path, hopefully you’ve interacted with them enough to help them find a better fit for their skills. If not, and it’s an uncomfortable conversation for you, I’d be happy to engage with individuals to help them find a better place for their energy and passion. I’m sure mhoye and others would also be glad to help out with the hard stuff.
Are we really in a state where we’re target-constrained? My experience with Mozilla is that there’s effectively infinite possible work. There’s 312 firefox-backlog+ bugs assigned to nobody@, and that number will grow. People should take a bug once they have a patch (though they can comment to say they’re trying to produce a patch.)
All that said, I’d consider it a high quality problem to have so many people wanting to fix bugs that we’re out of useful work to give them.
Having done this a lot, the following three things, in order, are pretty key to encouraging and including talented people:
* Review turnaround.
If we can tag individuals as “in mentoring” (basically “newbie who’s cleared the #introduction steps”) and then prioritize responses to bugs/patches for that group, we’re winning. Think about how hard it is when reviews get delayed and you have to work on other things, and then come back and fix stuff. Now imagine if the context you’re swapping in is “all of Mozilla tech” and you’ve got a day job or school. Reviews and feedback requests from people in mentoring should trump everyone else short of a chemspill.
I’d suggest “time from taking bug to bug landed and in Nightly” is the key pseudo-metric here. Everyone wants that sense of accomplishment, and we should help them get it quickly.
* Integration into the group.
Get them on IRC, get them interacting with the rest of the team so they feel like they’re part of the project, not outsiders. Have conversations, even somewhat crunchy ones, somewhere they can be a part of it.
* Show them the love.
Swag, work week invites, delivery of tasty things, summits and MozCamps. These are all ways we should people they’re appreciated and important.
These are all hard, and require effort and diligence to get right.
|Re: Contributor pathways, engagement points and bug mentoring||Michael Connor||4/21/14 3:45 PM|
On Apr 21, 2014, at 3:42 PM, Gijs Kruitbosch <gijskru...@gmail.com> wrote:I’ve been inactive in mainline dev for a fair while. (I keep filing the points on my head away, but they grow back.) Largely I’m thinking 2005-2010 timeframe. I’m also including things like ‘interns’ in this count. Not every intern is a rock star, even perfect GPA kids from top schools. It happens, and it’s hard to accurately predict.
I’ve seen little evidence that we aren’t attracting the same caliber of people. It’s a bit hard to prove either way, of course.
|Re: How to best engage with and invest time in new/beginning contributors (was: Re: Contributor pathways, engagement points and bug mentoring)||Michael Connor||4/21/14 3:53 PM|
I think this is a judgement call, for the most part. It’s probably worth checking
If we do, we’re erring all the way on the side of eliminating false positives. If we didn’t help people without an established track record, we’d be missing a lot of people. Including a number of people who’ve commented on this thread. I’m not convinced that’s a path toward winning.
|Re: Contributor pathways, engagement points and bug mentoring||Chris Peterson||4/21/14 5:04 PM|
On 4/21/14, 3:45 PM, Michael Connor wrote:Mike, do we have statistics on how contributors find us? For example,
how many come through Bugs Ahoy, school projects, or fixing a bug that
|Re: Contributor pathways, engagement points and bug mentoring||Karl Dubost||4/21/14 5:34 PM|
That would mean you have been able in advance to define what is "success" and what is "failure". My experience (not at Mozilla), but at W3C in terms of contributors to the W3C validators, spec reviews, etc. is that the return on investment is far to be even and success is a very difficult thing to assess.
* There are people who have completed the one task that they wanted to do and will never come back. It was successful because it was their initial goal.
* There are people who understood, and will not participate, BUT, WHEN the time permits in the future, will contribute actively.
* There are people where the mentoring didn't lead to direct contributions but to them being strong advocates for the project leading other people to come. They came to learn how it was happening and in return encourage others to do.
I think in all these discussions we are having we forget about our own attitudes with regards to other projects in life, sports club, etc. We met people who gave us a bit of our time to understand an area, … sometimes we pushed further, sometimes we gave up.
It. is. ok.
The people who mentored us, shared, etc. did it out of love for what they are doing, not for productivity marks ;)
Now it is entirely possible (and acceptable) that someone doesn't want to do that :) No issue.
Karl Dubost, Mozilla
|Re: Contributor pathways, engagement points and bug mentoring||Gavin Sharp||4/21/14 6:27 PM|
On Mon, Apr 21, 2014 at 5:34 PM, Karl Dubost <kdu...@mozilla.com> wrote:No, I think the point I'm making is independent of whatever definitions you
use for those words (as long as you recognize that both are possible
outcomes). I agree with you that "success" can mean many different things,
and we shouldn't define it too narrowly.
|Re: Contributor pathways, engagement points and bug mentoring||Ehsan Akhgari||4/21/14 7:55 PM|
On 2014-04-21, 4:02 PM, Gavin Sharp wrote:
> expecting a 100% "success" rate - but there's no getting around the factYeah, I see this problem. But can't we try to address this by educating
the mentors about this?
FWIW I'd be curious to see if we can gather data about the experiences
of the people who have mentored new contributors in the past. That way
we can approach this problem with some concrete data.
|Re: How to best engage with and invest time in new/beginning contributors||Ehsan Akhgari||4/21/14 8:09 PM|
On 2014-04-21, 6:42 PM, Michael Connor wrote:This. I'd actually go as far as saying that as far as a potential
future contributor is concerned, the review turn-around time is a sign
of respect for their contribution from the project's part.
As a personal rule, if I think that I won't be able to handle my review
in a given day, I always prioritize review requests from people I don't
know. Many of them turn out to be new contributors (whether paid or not.)
Yes, delayed gratification rarely wins.
I'd like to add a related entry:
* Help them make progress.
Often after I help someone lands his first patch at Mozilla, I politely
indicate to them that I'd be available to help the take on a second bug
if they're interested. If they say they are, I'll usually suggest some
other bugs to them to look at, and also try to make it clear that if
they have their own idea of what they would want to work on next that's
There's also some psychology involved, watch this video for example
<https://www.youtube.com/watch?v=u6XAPnuFjJc> which summarizes a lot of
good points about how to motivate people.
|Re: How to best engage with and invest time in new/beginning contributors||Mike Hoye||4/22/14 5:19 AM|
On 2014-04-21, 11:09 PM, Ehsan Akhgari wrote:This is a measurably important thing; we learned last year at MSR2013
that speed of response time to a patch is strongly correlated to
retention. If we turn around a review in the first day, we're quite
likely to see a second patch from that contributor. If we wait a week,
that contributor is very likely to move on.
|Re: Contributor pathways, engagement points and bug mentoring||Michael Connor||4/22/14 7:59 AM|
On Apr 21, 2014, at 4:02 PM, Gavin Sharp <ga...@gavinsharp.com> wrote:If we're seeing a 100% failure rate (or even 90%), we should be doing some sort of group post-mortem to understand why it’s that bad. That’s not a number I’ve heard previously. If it’s just some people, that would suggest the mentors need help and coaching. If it’s everyone, we’ve probably got deeper issues than how we identify and classify good bugs.
It’d be absurd to take this to extremes, and that’s not my argument. My point is that in deciding how far we turn the dial on “effort we expend to get a contributor up to speed” we’re either going to over-invest in people who aren’t worth it, or under-invest in people who are worth it. Do we want more useful contributors, at the expense of more wasted time, or should we run leaner and only invest time in the obviously awesome people? How do we measure and optimize our returns here?
|Re: Contributor pathways, engagement points and bug mentoring||Mike Hoye||4/22/14 8:01 AM|
On 2014-04-21, 10:55 PM, Ehsan Akhgari wrote:
> On 2014-04-21, 4:02 PM, Gavin Sharp wrote:
> Yeah, I see this problem. But can't we try to address this byI interviewed a dozen or so engineers last year; not a huge sample, but
I for the few common points among them I think reasonably representative.
The highlights - lowlights, really? - of that process were that everyone
agrees that helping new contributors grow is important and wants to do
more, but developers generally feel equally bad about three things:
- cutting off aspiring contributors when it becomes clear they're
thrashing/out of their depth/getting nowhere etc.
- having their mentoring relationship drag on unproductively at
painful length, and
- not having a clear sense of where mentoring falls onto their list
of priorities and quarterly goals.
Everyone I spoke to made some variation of these three points as being
the most difficult things about mentoring contributors, so my hope for
addressing them is:
- To make it clear to new contributors, per-bug, what Mozilla's
expectations are as far as readiness and experience are, and what we
think will make for a successful contribution
- To set out some guidelines for mentors (check in every week, show some
prior work, etc...) about how to make that relationship work and how to
positively direct a contributor towards a better fit when it's not
- To advocate that engineers be given quarterly goals around engagement
and the time to pursue them successfully.
That last thing I'm beating the drums for is basically "one mentored bug
in-progress per product cycle per engineer", which I think given some
better context - clear next steps for both parties, good alternative
plans and a clear sense that even with the best resources and intentions
sometimes it won't work and that's fine, it should be a better and more
productive experience for all concerned.
|Re: Contributor pathways, engagement points and bug mentoring||Boris Zbarsky||4/22/14 8:51 AM|
On 4/22/14, 10:59 AM, Michael Connor wrote:Anecdote, not data: a lot of people I've dealt with recently seem to be
showing up trying to contribute because they're university students with
a "contribute a patch to an open-source project" homework assignment.
This is becoming reasonably common in some countries (just like the
"report a bug in bugzilla" homework assignment that causes useless bug
This tends to not work all that great even for the first patch, and
tends to have near-0 post-first-patch retention....
|Re: Contributor pathways, engagement points and bug mentoring||Chris Hofmann||4/22/14 9:15 AM|
It makes sense to have a different path and different set of success
metrics for these kind of contributors.
First thing might be to try and trace back to the institutions that are
encouraging/requiring this kind of work. Having a list of contacts
there could help to improve the communication and make these limited
contributions more successful.
Building a set of Teaching Assistants within those organizations or
within Mozilla could keep the regular contributors/mentors building new
contributors that plan to stick around for more than just one "assignment."
Again, its my experience that the ratio's look something like this.
Talk to 1000 people. Nine or Ten show some interest in contributing.
One or two actually contribute something. After awhile this feels like
100% failure rate, but those 1 or 2 are critical to building and
sustaining the project going forward. The lessons here are about
setting the right expectations. Mentoring and expanding the project
with more contributors is really hard work. That doesn't mean we should
not do it.
|Re: Contributor pathways, engagement points and bug mentoring||Valentin Gosu||4/22/14 10:22 AM|
As someone who suggested open-source contributions as bonus assignments to
students, I can confirm that doing so in bulk resulted in a very low
percentage of a second patch (less than 5%). However, with two of the ones
that contributed one initial patch, I was able to get in touch with, and
mentor for an entire summer. This resulted in several good and useful
patches, increased involvement, and one intern for the upcoming summer.
Some of the things I noted:
- contributor involvement is directly proportionate with the time you put
- reviews that take more than 3 days are a huge turn-off (especially for
new contributors, since they're doing one bug at a time)
- in-person meetings are really productive, and video meetings are better
- new contributors aren't used to asking questions. Very often they get
stuck on something, and then they go away.
- challenging bugs are usually more engaging than simple bugs, if there is
good mentor to point them in the right direction.
While I agree that 1 or 2 people out of 10 who show an interest is fine, I
think we could do better. If we could set up a small group (reps maybe?)
tasked with reaching out to new contributors and helping them to get
started, I think our success rate would be much higher.
|Re: Contributor pathways, engagement points and bug mentoring||Mike Hoye||4/22/14 11:16 AM|
On 2014-04-22, 12:15 PM, Chris Hofmann wrote:
>> Anecdote, not data: a lot of people I've dealt with recently seem toWe're not really equipped to treat these contributors differently at
this point, and I don't think that it would be a positive step if we
were. There's not a lot we can do about ill-considered curriculum
decisions that want to use us as free instructional labor by the time
the curriculum's been decided. What we really need is a way to be in the
room when those decisions are getting made.
I think that we need to engage with the institutions pursuing these
initiatives, rather than the individuals doing the filing. Right now we
don't really have a setup that's compatible with student projects -
single-contributor focus and two- or six-week cycles vs. small teams and
term-based cycles - and making new Dave Humphreys doesn't seem to be in
If a prof or institution is ready to collaborate with us, we should be
able - and in many cases already can - connect them with a reasonable
set of bugs, projects or goals, maybe even regionally relevant goals
with the Reps involvement. But I think that this is the responsibility
of the larger Engagement team, but that beyond having a short list of
nice-to-have speculative-but-not-priority features, I don't think core
engineering specifically needs to invest a lot of effort into to this.
|Re: Contributor pathways, engagement points and bug mentoring||Gavin Sharp||4/22/14 11:33 AM|
On Tue, Apr 22, 2014 at 8:01 AM, Mike Hoye <mh...@mozilla.com> wrote:This sounds great.
|Re: Contributor pathways, engagement points and bug mentoring||Brandon Benvie||4/22/14 12:15 PM|
|Re: Contributor pathways, engagement points and bug mentoring||nhirata_bz||4/22/14 12:20 PM|
That sounds like a really cool notion. Should we explore it?
|Re: Contributor pathways, engagement points and bug mentoring||Mike Hoye||4/22/14 12:35 PM|
On 2014-04-22, 3:20 PM, Naoki Hirata wrote:Right now - right this week, in fact! - the engagement team is
developing a set of Mozcamp sessions to teach mentoring and
domain-specific advocacy and recruitment skills, with the goal of
levelling up contributors to mentors and evangelists.
We'll have more to report on that in a few days, so stay tuned!
|Re: Contributor pathways, engagement points and bug mentoring||Chris Hofmann||4/22/14 1:01 PM|
If we don't figure out a plan for these kind of contributions we are
setting ourselves up for one class of mentoring failures.
Mentor spends lots of time and effort helping a student along, then
student disappears after one bug. Mentor seeks ways to avoid that
including not doing the mentoring or fast reviews....
Yep, your right. That's one of the wonders of open source/free software
So we should look for easy ways to get in tune with what's going on.
Seems like asking a few new contributors where they are coming from,
what's there motivation, and if student what
institution-class-instructor sent them our way up front, instead of
after is an easy way to get a ahead of this a bit.
or just try and circulate some information (wiki page) about how to make
these kinds of contributions and participation more successful. Get
the instructors/professors to share their thoughts there...
>> I think that we need to engage with the institutions pursuing these initiatives, rather than the individuals doing the filing. Right now we don't really have a setup that's compatible with student projects - single-contributor focus and two- or six-week cycles vs. small teams and term-based cycles - and making new Dave Humphreys doesn't seem to be in the cards.agree that that "high touch" kind of effort is not in the cards. I'm
suggesting "lower touch" ways of communicating and guiding these kinds
>> If a prof or institution is ready to collaborate with us, we should be able - and in many cases already can - connect them with a reasonable set of bugs, projects or goals, maybe even regionally relevant goals with the Reps involvement.but that's not happening. do reps really know what to tell the profs?
could they know and understand what to tell them?
|Re: Contributor pathways, engagement points and bug mentoring||Mike Hoye||4/22/14 1:57 PM|
On 2014-04-22, 4:01 PM, Chris Hofmann wrote:
>>> We're not really equipped to treat these contributors differently atA contributor that completes one bug without coming back is not the best
possible outcome, but it is certainly not a failure.
But I stand by my claim: by the time we discern (if we manage to do that
at all) that this is a thing, it's too late to change the curriculum or
We can't make this engineering's problem; kids who've been set up to
fail by an instructor who didn't realize what they were asking for are
out of any scope we want to be a part of, and trying to address that
problem will be just heartbreak like turtles, all the way down.
The only current-term reaction that makes sense is what we're already
doing for individuals. The long game after that is for the engagement
team to reach out to the institution to try to help create an
achievable, relevant curriculum for the next wave of students.
|Re: Contributor pathways, engagement points and bug mentoring||fqu...@mozilla.com||4/23/14 6:48 AM|
On Sun, Mar 30, 2014 at 10:49 PM, Mike Hoye <mh...@mozilla.com> wrote:
> - The Mentoring doc spells out what makes a good first bug and a
> well-functioning mentoring relationship.
I wrote down my thoughts about mentoring Google Summer of Code students:
I think GSoC is an excellent opportunity to get new long term
contributors, but it requires good mentoring.
One of the key difference between what's already been discussed here
and the GSoC situation is that you don't have to worry about whether
the student will submit a second patch after the first one is
finished; but you instead need to motivate the student to stay around
and continue to contribute after the final payment is received.
|Re: Contributor pathways, engagement points and bug mentoring||Mike Hoye||4/23/14 7:36 AM|
Not yet. We've got a project underway called Baloo:
... with that as its goal, and we're connecting as many sites as we can
into Google Analytics, but we haven't tied the whole picture together yet.
I actually wrote "tied the whole picture together" there, like that's a
coherent sentence that will evoke a clear, comprehensible picture in
your imagination. Brain, what good are you, why are you even here.
Our data is incomplete but growing fast, and I'll have more to say about
|Re: How to best engage with and invest time in new/beginning contributors||Mike Hoye||4/23/14 8:59 AM|
On 2014-04-21, 6:53 PM, Michael Connor wrote:I've written up some best-practices - some even data-driven! - here:
But in short, contributors:
- Must have their dev environment set up and building successfully
before they take on a mentored bug,
- Might be asked to show some prior work or a tentative first run at a
patch before they can take the bug,
- Should check in once a week, so everyone can see things are
- Must respect their mentor's guidance with respect to patch
submission, review and direction towards a better-fitting opportunity if
Mentors, in turn:
- Should be judicious about who you agree to mentor,
- Must get back to mentorees' inquiries within 24 hours, and
- Should regularly evaluate if progress is being made, and redirect
the contributor to another opportunity if they've stalled out, and
- Should promptly deassign the bug as soon as they feel the
contributor has lost interest/disappeared.
"Radio silent for two weeks" is a pretty reasonable threshold for
pulling the trigger on that last part, I think, but as always these are
guidelines, not policies. No rule will ever be more important than the
mentor's best judgement.