Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Contributor pathways, engagement points and bug mentoring

161 views
Skip to first unread message

Mike Hoye

unread,
Mar 30, 2014, 4:49:37 PM3/30/14
to dev-pl...@lists.mozilla.org
Hi, everyone -

Like the title says, I've written up a couple of things about:

Code-specific contributor pathways:
https://wiki.mozilla.org/Contribute/Coding/Pathways

Engagement points: https://wiki.mozilla.org/Contribute/Coding/Engagement

and Mentoring / Good First Bugs:
https://wiki.mozilla.org/Contribute/Coding/Mentoring

... at some length, because I love me some big words and that's how I
roll. I'll spare you the details; they're there if you're interested. In
short:

- The Pathways doc outlines contribution tiers, how we're going to
measure engagement at every level, and some starting practices for
retention and reward. Most of it is generally of the form "We know it's
important to get back to new contributors promptly, so please let's do
that. When they finish a bug, thank them and suggest the next one."

- The Engagement doc lists the various ways that various people can put
interesting stuff in front of interested peoples' eyes. If I had to fit
this doc into a tweet, it would be "Choice paralysis is real, you don't
need to reinvent any wheels, we're here to help."

- The Mentoring doc spells out what makes a good first bug and a
well-functioning mentoring relationship. At the moment, most of our
good-first-bugs just aren't and a lot of contributors are getting
unintentionally pitched overboard -
https://bugzilla.mozilla.org/show_bug.cgi?id=989476 aims to fix one
such example - so I intend make auditing and automating those processes
a q2 goal.

I'd like to get us to contributor parity - the same number of paid
engineers as volunteer contributors getting patches in every product
cycle - within the next four releases. So I'm going to try to make that
happen.

Thanks.

- mhoye




Nick Alexander

unread,
Mar 30, 2014, 8:31:23 PM3/30/14
to dev-pl...@lists.mozilla.org
On 2014-03-30, 1:49 PM, Mike Hoye wrote:
> Hi, everyone -
>
> Like the title says, I've written up a couple of things about:

Just wanted to say that these are valuable (I specifically bookmarked
Pathways) and that they led me to all the curated content under
Contribute/. Thanks for spending time to write down concrete best
practices for how to help grow our Mozilla community.

Best,
Nick

Josh Matthews

unread,
Mar 30, 2014, 10:03:52 PM3/30/14
to
On 03/30/2014 04:49 PM, Mike Hoye wrote:
> - The Pathways doc outlines contribution tiers, how we're going to
> measure engagement at every level, and some starting practices for
> retention and reward. Most of it is generally of the form "We know it's
> important to get back to new contributors promptly, so please let's do
> that. When they finish a bug, thank them and suggest the next one."

One strategy I've started suggesting to mentors is to _offer_ to suggest
a next bug to work on. This allows the mentor to acknowledge the success
immediately, while giving them time to prepare an appropriate followup
if necessary.

Cheers,
Josh

Chris Peterson

unread,
Mar 31, 2014, 1:15:25 AM3/31/14
to
On 3/30/14, 1:49 PM, Mike Hoye wrote:
> I'd like to get us to contributor parity - the same number of paid
> engineers as volunteer contributors getting patches in every product
> cycle - within the next four releases. So I'm going to try to make that
> happen.

This is an exciting plan! What is our current ratio of patches from
employee vs volunteer contributors? How do you determine whether a patch
is from Mozilla employee? Many employees' Bugzilla accounts and patches
use their personal email addresses instead of mozilla.com addresses.


chris

Josh Matthews

unread,
Mar 31, 2014, 1:36:44 AM3/31/14
to
Employee status is derived from the list of email addresses obtained via
https://github.com/jdm/outrider/blob/master/generate_emails.py , which
scrapes from all email fields in the internal phonebook. I am aware of
very few inaccuracies, since most employees have been happy to update
their entry with addresses they use.

Cheers,
Josh

Mike Hoye

unread,
Mar 31, 2014, 11:25:49 AM3/31/14
to dev-pl...@lists.mozilla.org
On 2014-03-31, 1:15 AM, Chris Peterson wrote:
>
> This is an exciting plan! What is our current ratio of patches from
> employee vs volunteer contributors?
We're generally floating around 50% per rapid release now; I'll have
some better historical and trend data to share shortly.


- mhoye

David Rajchenbach-Teller

unread,
Apr 3, 2014, 7:09:05 AM4/3/14
to Josh Matthews, dev-pl...@lists.mozilla.org
At some point, we should start thinking about a contributor dashboard,
which would handle both Bugs Ahoy and "Find a next bug".

On 3/31/14 4:03 AM, Josh Matthews wrote:
> One strategy I've started suggesting to mentors is to _offer_ to suggest
> a next bug to work on. This allows the mentor to acknowledge the success
> immediately, while giving them time to prepare an appropriate followup
> if necessary.
>
> Cheers,
> Josh
> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning


--
David Rajchenbach-Teller, PhD
Performance Team, Mozilla

Gavin Sharp

unread,
Apr 6, 2014, 3:32:48 PM4/6/14
to Mike Hoye, dev. planning
Great stuff. A couple of points of feedback:

- https://wiki.mozilla.org/Contribute/Coding/Engagement#Mailing_lists

This should probably include firefox-dev?
(https://wiki.mozilla.org/Firefox/firefox-dev)

- https://wiki.mozilla.org/Contribute/Coding/Pathways

I think this is missing some important bug triage activities. Beyond
"filing" and "reproducing" and "creating a testcase", there's a lot of
value in "marking duplicates" or "confirming bugs" (and the
prerequisite step to those, "gaining editbugs privileges"). These are
how I got sucked in to Mozilla, and it's what eventually led me to
code contributions.

Gavin

On Sun, Mar 30, 2014 at 1:49 PM, Mike Hoye <mh...@mozilla.com> wrote:
> Hi, everyone -
>
> Like the title says, I've written up a couple of things about:
>
> Code-specific contributor pathways:
> https://wiki.mozilla.org/Contribute/Coding/Pathways
>
> Engagement points: https://wiki.mozilla.org/Contribute/Coding/Engagement
>
> and Mentoring / Good First Bugs:
> https://wiki.mozilla.org/Contribute/Coding/Mentoring
>
> ... at some length, because I love me some big words and that's how I roll.
> I'll spare you the details; they're there if you're interested. In short:
>
> - The Pathways doc outlines contribution tiers, how we're going to measure
> engagement at every level, and some starting practices for retention and
> reward. Most of it is generally of the form "We know it's important to get
> back to new contributors promptly, so please let's do that. When they finish
> a bug, thank them and suggest the next one."
>
> - The Engagement doc lists the various ways that various people can put
> interesting stuff in front of interested peoples' eyes. If I had to fit
> this doc into a tweet, it would be "Choice paralysis is real, you don't need
> to reinvent any wheels, we're here to help."
>
> - The Mentoring doc spells out what makes a good first bug and a
> well-functioning mentoring relationship. At the moment, most of our
> good-first-bugs just aren't and a lot of contributors are getting
> unintentionally pitched overboard -
> https://bugzilla.mozilla.org/show_bug.cgi?id=989476 aims to fix one such
> example - so I intend make auditing and automating those processes a q2
> goal.
>
> I'd like to get us to contributor parity - the same number of paid engineers
> as volunteer contributors getting patches in every product cycle - within
> the next four releases. So I'm going to try to make that happen.
>
> Thanks.
>
> - mhoye

Mike Hoye

unread,
Apr 7, 2014, 10:04:05 AM4/7/14
to dev. planning
On 2014-04-06, 3:32 PM, Gavin Sharp wrote:
> Great stuff. A couple of points of feedback:
>
> - https://wiki.mozilla.org/Contribute/Coding/Engagement#Mailing_lists
>
> This should probably include firefox-dev?
> (https://wiki.mozilla.org/Firefox/firefox-dev)
It should - there are a few missing fora that need to go in there,
including firefox-dev.
>
> - https://wiki.mozilla.org/Contribute/Coding/Pathways
>
> I think this is missing some important bug triage activities. Beyond
> "filing" and "reproducing" and "creating a testcase", there's a lot of
> value in "marking duplicates" or "confirming bugs" (and the
> prerequisite step to those, "gaining editbugs privileges"). These are
> how I got sucked in to Mozilla, and it's what eventually led me to
> code contributions.

A strong point, I'll add that.

- mhoye

cb...@mozilla.com

unread,
Apr 15, 2014, 7:41:55 AM4/15/14
to
I think the part about

" That a contributor is recognized for their efforts, even if this is a simple as thanks, and "

is super important.

I do now as one of the sheriffs some of the checkin-needed requests and it happens sometimes that i see a bug from a new contributor who submits his/her first patch that got reviewed and now landed in our codebase.

Doing a comment in bugzilla like "thanks for contributing and welcome to mozilla" is i guess a important thing to let them feel welcome and that their work is recognized!

So i think that (while its simple) a important thing to keep people engaged into mozilla.

Mike Hoye

unread,
Apr 15, 2014, 9:54:21 AM4/15/14
to dev-pl...@lists.mozilla.org
On 2014-03-31, 11:25 AM, Mike Hoye wrote:
> On 2014-03-31, 1:15 AM, Chris Peterson wrote:
>>
>> This is an exciting plan! What is our current ratio of patches from
>> employee vs volunteer contributors?
> We're generally floating around 50% per rapid release now; I'll have
> some better historical and trend data to share shortly.

Ah, yeah - to clear this up, this really means "we have half as many
non-employees contributing as employees per rapid-release". And that's
people - as you might imagine, the ratio in terms of patches contributed
is much lower.

Some good news, following on from this last two weeks, is that uptake on
@startmozilla seems to be trending in the right direction and new people
are still showing up to work on stuff. "New contributors" and
"contributor retention" are only proxy measurements for the high-value,
long-term contributors we really want, obviously, but as metrics go
they're non-terrible and and it looks like we can move the needles on
those gauges meaningfully in the near term.

Some bad news is that the cultural impedance between our reliance on
NeedInfo and new contributors arriving from "ask" cultures is worse than
I thought.

To sum up - by relying exclusively on needinfo to know what bugs to look
at, we're losing a lot of potential contributors who ask for permission
to participate in a bug comment, get a demoralizing silence in response
and leave. The forty or so instances of this I've found - and these are
just the ones I've found, mind you - all have that same pattern, and
possibly worse, they're almost all coming from markets - mostly India
and southeast Asia - where we really want to make a lot of new friends.

I've filed bug 989476 -
https://bugzilla.mozilla.org/show_bug.cgi?id=989476 - to mitigate
somewhat, the TL;DR being is that if you've agreed to be a mentor in a
bug, any comment from a "new to bugzilla" contributor will trip the
needinfo flag. If you've got strong feelings about that for or against,
that's where to register them.

- mhoye

Gavin Sharp

unread,
Apr 15, 2014, 5:29:28 PM4/15/14
to Mike Hoye, dev. planning
On Tue, Apr 15, 2014 at 6:54 AM, Mike Hoye <mh...@mozilla.com> wrote:
> Some bad news is that the cultural impedance between our reliance on
> NeedInfo and new contributors arriving from "ask" cultures is worse than I
> thought.
>
> To sum up - by relying exclusively on needinfo to know what bugs to look at,
> we're losing a lot of potential contributors who ask for permission to
> participate in a bug comment, get a demoralizing silence in response and
> leave. The forty or so instances of this I've found - and these are just the
> ones I've found, mind you - all have that same pattern, and possibly worse,
> they're almost all coming from markets - mostly India and southeast Asia -
> where we really want to make a lot of new friends.

Speaking based on personal experience, I think such comments tend to
be ignored because they very infrequently lead to a positive new
contributor interaction, and not because of any "reliance on
needinfo".

Feels like there's probably room to tackle this from the other angle -
i.e. teaching people they don't need to "ask first".

Gavin

Gijs Kruitbosch

unread,
Apr 15, 2014, 6:09:59 PM4/15/14
to
On 15/04/2014 22:29, Gavin Sharp wrote:
> Feels like there's probably room to tackle this from the other angle -
> i.e. teaching people they don't need to "ask first".

This.

And also, hard choices about what kind of engagement we do and how
selective we are about contributors that don't seem to "get" this or be
able to work without asking. And more asking. And more asking.

I've had a few experiences now where people responding to good first
bugs or mentored bugs had to be handheld to such a degree (ie,
essentially needed literal instructions along the line of "what exact,
character-by-character, code changes do I need to make to fix this
bug"), and even didn't manage to do what was asked ("please run these
tests with ./mach mochitest-browser foo" - "I've run them and they pass"
- push code, break tree because those exact tests fail).

This makes me reluctant to sign up to mentoring bugs, because this way,
bugs repeatedly take orders of magnitude more time from me than they
would have done if I'd fixed them myself, with no discernible advantage
to myself or the project (aforementioned types of contributors move on
to other bugs with the same method, taking just as much time from myself
or others, and/or disappear).

To be clear, I understand that we need to invest in enlarging the
community, and that this takes time and effort, and that it's expected
and OK that the first (few) bug(s) that someone fixes take more time
from the mentor than fixing it would have done had they done it
themselves, and that not everyone sticks around. And that's fine. But
the degree to which I've seen people's (not just my own) time being
eaten up by certain aspiring contributors was very demotivating, and I
wonder what, if anything, we can/are do(ing) to fix this.

~ Gijs

Ehsan Akhgari

unread,
Apr 15, 2014, 6:17:17 PM4/15/14
to Gijs Kruitbosch, dev-pl...@lists.mozilla.org
On 2014-04-15, 6:09 PM, Gijs Kruitbosch wrote:
> On 15/04/2014 22:29, Gavin Sharp wrote:
>> Feels like there's probably room to tackle this from the other angle -
>> i.e. teaching people they don't need to "ask first".
>
> This.
>
> And also, hard choices about what kind of engagement we do and how
> selective we are about contributors that don't seem to "get" this or be
> able to work without asking. And more asking. And more asking.

That is very unfair to people who are coming from cultures where jumping
in the middle of something without asking for permission first is
considered a very impolite thing to do. We're not talking about typical
Western cultures that we mostly operate under.

> I've had a few experiences now where people responding to good first
> bugs or mentored bugs had to be handheld to such a degree (ie,
> essentially needed literal instructions along the line of "what exact,
> character-by-character, code changes do I need to make to fix this
> bug"), and even didn't manage to do what was asked ("please run these
> tests with ./mach mochitest-browser foo" - "I've run them and they pass"
> - push code, break tree because those exact tests fail).

I think this is a separate issue. Not every contributor is going to be
as savvy as all others.

> This makes me reluctant to sign up to mentoring bugs, because this way,
> bugs repeatedly take orders of magnitude more time from me than they
> would have done if I'd fixed them myself, with no discernible advantage
> to myself or the project (aforementioned types of contributors move on
> to other bugs with the same method, taking just as much time from myself
> or others, and/or disappear).
>
> To be clear, I understand that we need to invest in enlarging the
> community, and that this takes time and effort, and that it's expected
> and OK that the first (few) bug(s) that someone fixes take more time
> from the mentor than fixing it would have done had they done it
> themselves, and that not everyone sticks around. And that's fine. But
> the degree to which I've seen people's (not just my own) time being
> eaten up by certain aspiring contributors was very demotivating, and I
> wonder what, if anything, we can/are do(ing) to fix this.

Speaking from my personal experience when I first started to contribute
to Mozilla, it took me a *month* to be able to build the code base
locally. During that time a couple of individuals continued to try
their best to help me along the way. And without that, I would have
never stuck around the project.

The goal of attracting new contributors is not to help our engineers get
stuff done faster. It is to attract new talent to the project who can
help drive it forward in ways that no single contributor can. It's OK
if someone doesn't have time to help with that (I rarely have enough
time for that these days myself) but let's not forget the bigger picture
here.

Cheers,
Ehsan

Mike Connor

unread,
Apr 15, 2014, 6:36:37 PM4/15/14
to Ehsan Akhgari, Gijs Kruitbosch, dev-pl...@lists.mozilla.org
To add to this: a great many of us "didn't get it" early on. Mozilla code
is hard and daunting at first, and there are many ways to get lost. If I
didn't have bz/dwitte/timeless patiently answering questions, I'd have
bounced in 2003 without ever getting a build working, let alone
accomplishing anything I've done in the ensuing decade. Without outing
people on this list, there are many established Firefox hackers who needed
a lot of support in the early days. If we aren't providing that support
and guidance, we'll miss out.

Is it fun? Not always. Do we all have other things we could be doing?
Sure. Is it an essential part of how we bring people in (and how we learn
to be better ourselves)? Absolutely.

-- Mike

-----Original Message-----
From: dev-planning
[mailto:dev-planning-bounces+mconnor=mozil...@lists.mozilla.org] On
Behalf Of Ehsan Akhgari
Sent: April 15, 2014 6:17 PM
To: Gijs Kruitbosch; dev-pl...@lists.mozilla.org
Subject: Re: Contributor pathways, engagement points and bug mentoring

[Mike Connor] <snip>

Speaking from my personal experience when I first started to contribute to
Mozilla, it took me a *month* to be able to build the code base locally.
During that time a couple of individuals continued to try their best to
help me along the way. And without that, I would have never stuck around
the project.

The goal of attracting new contributors is not to help our engineers get
stuff done faster. It is to attract new talent to the project who can
help drive it forward in ways that no single contributor can. It's OK if
someone doesn't have time to help with that (I rarely have enough time for
that these days myself) but let's not forget the bigger picture here.

Cheers,
Ehsan

Gijs Kruitbosch

unread,
Apr 15, 2014, 6:58:32 PM4/15/14
to Mike Connor, Ehsan Akhgari
It seems my message wasn't clear, although I sort of saw this response
coming and tried to pre-empt it - evidently I failed.

I'm not complaining about people generally needing, and asking for, help.

9-odd years ago, when I started to contribute as a volunteer, I also
needed a lot of help. Heck, to "out" myself here: I needed help
understanding some of the programming language at issue (nobody had ever
told me String.replace() could take a function as the second argument!
(and JS was different from VBScript (yes...) in surprising ways)).
If/when I fix the occasional C++/objc bug, I need help with some of our
internals (and sometimes the language - clang helped a lot because the
error messages are so much better), still.

But needing help and needing literally all the things handed to you on a
silver platter are not the same.

I actually think we should be spending more time than we are on engaging
with new contributors (and it's something I plan on doing now that
Australis is finally being released). However, I also think we should be
careful about where and how we spend our efforts.

The time all of us, collectively, can spend on helping new contributors
up to speed is a limited resource. It would be nice if we could optimize
its usage better than we currently do.

Perhaps I (and one or two other people who mentored similar contributors
and with whom I've discussed this when these particular contributors
were active) were the only people affected, and the issue I'm bringing
up is such a minority case we don't need to worry about it. I hope so!
But if not, we should figure out how/where to draw a line.

It's never felt right to me to say "no" to someone who wants to own a
bug, but if previous evidence strongly suggests they don't stand a
chance of fixing them and will "occupy" the mentored bug when we only
have a limited number of them (seriously,
http://www.joshmatthews.net/bugsahoy/?ff=1&unowned=1 has very few bugs
that have never been (unsuccessfully) owned, so bugs with the right
difficulty level for first contributors are clearly hard to come by**),
then I would love to have a better way to respond than letting the
inevitable happen when I do assign them the bug.

~ Gijs

** which probably means we should also think harder on the fx desktop
team about which bugs could be such bugs, and how to make more of them!
Many of those bugs are devtools bugs, and there doesn't appear to be a
way to filter those as compared to the rest of Fx desktop; Without data,
I do believe the devtools team has been doing a better job in terms of
mentored bugs than we have.

Gavin Sharp

unread,
Apr 15, 2014, 6:59:53 PM4/15/14
to Mike Connor, Ehsan Akhgari, dev. planning, Gijs Kruitbosch
Straw man alert: No one is arguing for "not providing support and guidance".

Nor did I intend to suggest that "teaching people not to ask" is the
only option - I very explicitly mentioned that this should be tackled
from both angles.

My point (and I think Gijs' as well) is just that not all people who
express a desire to contribute can be successful in doing so, for
various reasons, and "respond more to people's comments" on its own is
unlikely to be a cost-effective or scalable solution. We need to have
systematic solutions for avoiding expending large amounts of effort
for minimal return, and efficient ways to direct potential
contributors to the right opportunities.

I know mhoye has given those problems lots of thought, so this isn't
news to him, but it felt worth clarifying in the context of this
thread.

Gavin

Trevor Saunders

unread,
Apr 15, 2014, 7:08:32 PM4/15/14
to dev-pl...@lists.mozilla.org
On Tue, Apr 15, 2014 at 11:58:32PM +0100, Gijs Kruitbosch wrote:
> It seems my message wasn't clear, although I sort of saw this response
> coming and tried to pre-empt it - evidently I failed.
>
> I'm not complaining about people generally needing, and asking for, help.
>
> 9-odd years ago, when I started to contribute as a volunteer, I also needed
> a lot of help. Heck, to "out" myself here: I needed help understanding some
> of the programming language at issue (nobody had ever told me
> String.replace() could take a function as the second argument! (and JS was
> different from VBScript (yes...) in surprising ways)). If/when I fix the
> occasional C++/objc bug, I need help with some of our internals (and
> sometimes the language - clang helped a lot because the error messages are
> so much better), still.
>
> But needing help and needing literally all the things handed to you on a
> silver platter are not the same.

yes, this

> I actually think we should be spending more time than we are on engaging
> with new contributors (and it's something I plan on doing now that Australis
> is finally being released). However, I also think we should be careful about
> where and how we spend our efforts.
>
> The time all of us, collectively, can spend on helping new contributors up
> to speed is a limited resource. It would be nice if we could optimize its
> usage better than we currently do.
>
> Perhaps I (and one or two other people who mentored similar contributors and
> with whom I've discussed this when these particular contributors were
> active) were the only people affected, and the issue I'm bringing up is such
> a minority case we don't need to worry about it. I hope so! But if not, we
> should figure out how/where to draw a line.

I've certainly seen this type of person too.

> It's never felt right to me to say "no" to someone who wants to own a bug,
> but if previous evidence strongly suggests they don't stand a chance of
> fixing them and will "occupy" the mentored bug when we only have a limited
> number of them (seriously,
> http://www.joshmatthews.net/bugsahoy/?ff=1&unowned=1 has very few bugs that
> have never been (unsuccessfully) owned, so bugs with the right difficulty
> level for first contributors are clearly hard to come by**), then I would
> love to have a better way to respond than letting the inevitable happen when
> I do assign them the bug.

yes please. Also if we do better at sending contributors who have no
hope of fixing the bug someplace else I hope we'll spend more time
helping people who are or at least will be useful in the future, and be
less frustrated while we do it.

Trev
signature.asc

Ehsan Akhgari

unread,
Apr 15, 2014, 7:30:50 PM4/15/14
to Trevor Saunders, dev-pl...@lists.mozilla.org
It seems we're mostly in violent agreement!

On 2014-04-15, 7:08 PM, Trevor Saunders wrote:
> On Tue, Apr 15, 2014 at 11:58:32PM +0100, Gijs Kruitbosch wrote:
>> It seems my message wasn't clear, although I sort of saw this response
>> coming and tried to pre-empt it - evidently I failed.
>>
>> I'm not complaining about people generally needing, and asking for, help.
>>
>> 9-odd years ago, when I started to contribute as a volunteer, I also needed
>> a lot of help. Heck, to "out" myself here: I needed help understanding some
>> of the programming language at issue (nobody had ever told me
>> String.replace() could take a function as the second argument! (and JS was
>> different from VBScript (yes...) in surprising ways)). If/when I fix the
>> occasional C++/objc bug, I need help with some of our internals (and
>> sometimes the language - clang helped a lot because the error messages are
>> so much better), still.
>>
>> But needing help and needing literally all the things handed to you on a
>> silver platter are not the same.
>
> yes, this
>
>> I actually think we should be spending more time than we are on engaging
>> with new contributors (and it's something I plan on doing now that Australis
>> is finally being released). However, I also think we should be careful about
>> where and how we spend our efforts.
>>
>> The time all of us, collectively, can spend on helping new contributors up
>> to speed is a limited resource. It would be nice if we could optimize its
>> usage better than we currently do.
>>
>> Perhaps I (and one or two other people who mentored similar contributors and
>> with whom I've discussed this when these particular contributors were
>> active) were the only people affected, and the issue I'm bringing up is such
>> a minority case we don't need to worry about it. I hope so! But if not, we
>> should figure out how/where to draw a line.
>
> I've certainly seen this type of person too.
>
>> It's never felt right to me to say "no" to someone who wants to own a bug,
>> but if previous evidence strongly suggests they don't stand a chance of
>> fixing them and will "occupy" the mentored bug when we only have a limited
>> number of them (seriously,
>> http://www.joshmatthews.net/bugsahoy/?ff=1&unowned=1 has very few bugs that
>> have never been (unsuccessfully) owned, so bugs with the right difficulty
>> level for first contributors are clearly hard to come by**), then I would
>> love to have a better way to respond than letting the inevitable happen when
>> I do assign them the bug.
>
> yes please. Also if we do better at sending contributors who have no
> hope of fixing the bug someplace else I hope we'll spend more time
> helping people who are or at least will be useful in the future, and be
> less frustrated while we do it.

Yeah agreed. I think it's totally OK to suggest different things for
those types of contributors to spend their time on if you believe their
contribution won't get anywhere based on past experience. We have quite
a bit of non-technical ways of contributing!

Mike Hoye

unread,
Apr 15, 2014, 10:18:03 PM4/15/14
to dev-pl...@lists.mozilla.org
On 2014-04-15, 6:59 PM, Gavin Sharp wrote:
>
> My point (and I think Gijs' as well) is just that not all people who
> express a desire to contribute can be successful in doing so, for
> various reasons, and "respond more to people's comments" on its own is
> unlikely to be a cost-effective or scalable solution. We need to have
> systematic solutions for avoiding expending large amounts of effort
> for minimal return, and efficient ways to direct potential
> contributors to the right opportunities.
>
> I know mhoye has given those problems lots of thought, so this isn't
> news to him, but it felt worth clarifying in the context of this
> thread.
There's a lot to unpack here.

There's no question that there will be some people who will not be
effective contributors after any amount of effort by any number of
established Mozillians. It's unfortunate, but those people exist and
nobody who takes on a mentoring role should feel bad or hesitate for a
second to say somebody isn't ready, or needs to show some work or
acquire some more basic skills before they can proceed. Mentoring isn't
doing the work for someone, and if that's what it's turning into,
absolutely try to direct that contributor to somewhere they might be
more successful. We have some great introductory and non-code
contribution channels now - #introduction, SuMo, QA, Localization, the
Reps, a bunch of others; the systematic solution I'm hoping we can adopt
in the near term is "know the other engagement pathways and send people
to them as soon as it occurs to you that you should."

Nevertheless, I don't think that the possibility that somebody _might_
be a time-sink means we can leave them hanging,

I'm making a very narrow claim, here. All the data I've got says that
these are not people who are expressing a desire to contribute and then
failing to deliver; these are people who are expressing a desire to
contribute and then inadvertently being ignored. I know the solution
I've proposed isn't all that pretty, but I'm looking at a set of
mismatched expectations between how people interact with a service we
own and control outright and how people interact with other people in
about three-fifths of the entire world. In that context an inelegant
solution we can push to production over the weekend looks like a pretty
sweet deal.

I agree that responding to more people's comments on its own isn't
cost-effective or scaleable, but getting new contributors to the point
where they can mentor the next generation of contributors themselves,
that scales like crazy. In fact, until late in the industrial
revolution, that was _the only thing that had ever_ scaled. And I don't
see a way to get from here to a million Mozillians that doesn't involve
both talking to lots of new people and leveling up the people we have.

If we get to the point that just talking to the number of new
contributors who want to fix bugs becomes an unsupportable burden for
Mozilla's engineers we will absolutely have to figure out how to address
that, but from up here in the cheap seats I bet that looks an awful lot
like winning.


- mhoye

Ehsan Akhgari

unread,
Apr 16, 2014, 11:53:42 AM4/16/14
to Mike Hoye, dev-planning@lists.mozilla.org planning
On Tue, Apr 15, 2014 at 10:18 PM, Mike Hoye <mh...@mozilla.com> wrote:

> I'm making a very narrow claim, here. All the data I've got says that
> these are not people who are expressing a desire to contribute and then
> failing to deliver; these are people who are expressing a desire to
> contribute and then inadvertently being ignored. I know the solution I've
> proposed isn't all that pretty, but I'm looking at a set of mismatched
> expectations between how people interact with a service we own and control
> outright and how people interact with other people in about three-fifths of
> the entire world. In that context an inelegant solution we can push to
> production over the weekend looks like a pretty sweet deal.
>
> I agree that responding to more people's comments on its own isn't
> cost-effective or scaleable, but getting new contributors to the point
> where they can mentor the next generation of contributors themselves, that
> scales like crazy. In fact, until late in the industrial revolution, that
> was _the only thing that had ever_ scaled. And I don't see a way to get
> from here to a million Mozillians that doesn't involve both talking to lots
> of new people and leveling up the people we have.
>

To get back to the original purpose of this thread, I actually question
Gavin's assertion about the cost effectiveness and scalability of answering
"yes please, you can work on this bug". As someone who has worked in two
very different cultures, I think it's very easy to dismiss these types of
asks before working on a bug as an indicator that the requestor is not
serious etc but that is far from the truth, they are just trying to be
polite within the cultural norms that they are used to.


> If we get to the point that just talking to the number of new contributors
> who want to fix bugs becomes an unsupportable burden for Mozilla's
> engineers we will absolutely have to figure out how to address that, but
> from up here in the cheap seats I bet that looks an awful lot like winning.
>

That is a great problem to have, and we are *far* from it.

--
Ehsan
<http://ehsanakhgari.org/>

Robert Kaiser

unread,
Apr 16, 2014, 12:31:24 PM4/16/14
to
Gijs Kruitbosch schrieb:
> ** which probably means we should also think harder on the fx desktop
> team about which bugs could be such bugs, and how to make more of them!

That's surely a good idea!

KaiRo

Benjamin Smedberg

unread,
Apr 16, 2014, 3:29:54 PM4/16/14
to Josh Matthews, dev-pl...@lists.mozilla.org
On 4/16/2014 3:23 PM, Josh Matthews wrote:
>
> There is also nothing wrong with unassigning someone who stops
> responding (a needinfo with a week's leniency seems polite to me).

The recommendation at
https://wiki.mozilla.org/Contribute/Coding/Mentoring is that we don't
assign bugs to new contributors until they've produced the patch. So the
response to "I'd like to work on this bug, please assign it to me"
should for new contributors be "Please feel free to work on this bug. We
don't assign bugs to new contributors until they have created a patch,
but please feel free to ask for help here as bug comments."

--BDS

Josh Matthews

unread,
Apr 16, 2014, 3:23:11 PM4/16/14
to
On 04/16/2014 02:58 PM, Chris Peterson wrote:
> On 4/16/14, 11:01 AM, Gavin Sharp wrote:
>> But, my personal experience has shown that a high percentage of the
>> people who ask "can you assign this bug to me" or some variant of it,
>> aren't actually ready to have the bug assigned to them. I have
>> assigned bugs, and spent time providing helpful comments, only to get
>> no response or to discover that the person is so far from being able
>> to contribute (for various reasons) that it's not an effective use of
>> my time to help them with that bug. Those experiences are what led to
>> me "ignore" such requests, for better or worse, and the needinfo
>> proposal isn't going to address that problem.
>
> Maybe we can craft some polite canned responses to new contributor's bug
> questions. If they have not met some minimum requirements like "I have
> successfully built Firefox and created a test patch", then we can direct
> them from Bugzilla to #introduction and the wiki.
>
>
> chris

Exactly. This is the role of #introduction; mentors should not feel
responsible for solving build environment issues that are not a direct
result of changes made in order to solve the bug under discussion. Off
the top of my head:
"Have you compiled Firefox from source? If not, please see
https://developer.mozilla.org/en/Simple_Firefox_build and direct further
questions to #introduction on http://irc.mozilla.org. When you a build
environment prepared, please feel free to start work on this bug
immediately, and I will try to answer any specific questions you have
about this task."

There is also nothing wrong with unassigning someone who stops
responding (a needinfo with a week's leniency seems polite to me).

Cheers,
Josh

Gavin Sharp

unread,
Apr 16, 2014, 2:01:57 PM4/16/14
to Ehsan Akhgari, dev-planning@lists.mozilla.org planning, Mike Hoye
On Wed, Apr 16, 2014 at 8:53 AM, Ehsan Akhgari <ehsan....@gmail.com> wrote:
> To get back to the original purpose of this thread, I actually question
> Gavin's assertion about the cost effectiveness and scalability of answering
> "yes please, you can work on this bug". As someone who has worked in two
> very different cultures, I think it's very easy to dismiss these types of
> asks before working on a bug as an indicator that the requestor is not
> serious etc but that is far from the truth, they are just trying to be
> polite within the cultural norms that they are used to.

I think you misunderstood my claim.

If all someone needs is a "yes please, go right ahead! Here are some
tips:", we should definitely provide it to them. If adding needinfos
helps with that, that's great, and I don't object to it.

But, my personal experience has shown that a high percentage of the
people who ask "can you assign this bug to me" or some variant of it,
aren't actually ready to have the bug assigned to them. I have
assigned bugs, and spent time providing helpful comments, only to get
no response or to discover that the person is so far from being able
to contribute (for various reasons) that it's not an effective use of
my time to help them with that bug. Those experiences are what led to
me "ignore" such requests, for better or worse, and the needinfo
proposal isn't going to address that problem.

So we need to tackle both sides of the problem. That's all I'm saying.

Gavin

Chris Peterson

unread,
Apr 16, 2014, 2:58:37 PM4/16/14
to
On 4/16/14, 11:01 AM, Gavin Sharp wrote:
> But, my personal experience has shown that a high percentage of the
> people who ask "can you assign this bug to me" or some variant of it,
> aren't actually ready to have the bug assigned to them. I have
> assigned bugs, and spent time providing helpful comments, only to get
> no response or to discover that the person is so far from being able
> to contribute (for various reasons) that it's not an effective use of
> my time to help them with that bug. Those experiences are what led to
> me "ignore" such requests, for better or worse, and the needinfo
> proposal isn't going to address that problem.

Michael Connor

unread,
Apr 21, 2014, 10:24:41 AM4/21/14
to Gavin Sharp, Ehsan Akhgari, dev-planning@lists.mozilla.org planning, Gijs Kruitbosch

On Apr 15, 2014, at 6:59 PM, Gavin Sharp <ga...@gavinsharp.com> wrote:

> We need to have
> systematic solutions for avoiding expending large amounts of effort
> for minimal return, and efficient ways to direct potential
> contributors to the right opportunities.

I think this gets held up a lot, but usually in the context of people arguing that we should make calls quicker on people to avoid sinking too much effort into an individual that won’t deliver a good enough return on that investment. My point is that we often can’t know, without investing a significant amount of time, whether someone will be an important contributor over the long run.

Largely, I think we worry a lot about the false positive case (when we invest too much in someone who wasn’t worth it), and don’t spend as much time thinking about the false negative case (when we fail to invest enough time in someone who would be an important contributor). Perhaps more importantly, I think we value the “lost” time for the false positive far more than we value the contributions we’re losing from the false negative case. That’s natural, but I think we can and should do better.

— Mike

Gijs Kruitbosch

unread,
Apr 21, 2014, 10:41:07 AM4/21/14
to Michael Connor, Gavin Sharp, Ehsan Akhgari
On 21/04/2014 15:24, Michael Connor wrote:
>
> On Apr 15, 2014, at 6:59 PM, Gavin Sharp <ga...@gavinsharp.com> wrote:
>
>> We need to have
>> systematic solutions for avoiding expending large amounts of effort
>> for minimal return, and efficient ways to direct potential
>> contributors to the right opportunities.
>
> I think this gets held up a lot, but usually in the context of people
> arguing that we should make calls quicker on people to avoid sinking too
> much effort into an individual that won’t deliver a good enough return
> on that investment. My point is that we often can’t know, without
> investing a significant amount of time, whether someone will be an
> important contributor over the long run.

I didn't mean to hold this up in terms of "we need quicker calls" but
more in terms of "how do we communicate such a call". I find it hard to
say "no" generally, and even harder in these circumstances.

However, I think that the below point is more important:

> Largely, I think we worry a lot about the false positive case (when we
> invest too much in someone who wasn’t worth it), and don’t spend as much
> time thinking about the false negative case (when we fail to invest
> enough time in someone who would be an important contributor). Perhaps
> more importantly, I think we value the “lost” time for the false
> positive far more than we value the contributions we’re losing from the
> false negative case. That’s natural, but I think we can and should do
> better.

Two points here:
1) as I pointed out, the false positive and false negative cases are
linked in that supplies are (relatively) low, and more than a few of the
bugs in http://www.joshmatthews.net/bugsahoy/?ff=1&unowned=1 were at one
point owned and went unfixed. All the time in which bugs are assigned to
people who don't fix them is time they spend making it harder for the
"right" people to find bugs to work on.

2) How can we reduce the number of false negatives? This is intended as
a genuine question - I would love to be able to help in that department.
Could we try to e.g. get team members to identify five mentored bugs a
week or something? Send swag to people for fixing mentored bugs? Invite
them for work weeks? Organize "fix mentored bugs" competitions/days?
Make it easier for people to fix bugs by improving the "effort curve"
for contributing? Something else? Do we (does Mike) have data on some of
this?

I'm sure Mike Hoye has lots of thoughts on this subject, but it's a
while away from his original post, so I'm threadsplitting (at least
subject-wise, ymmv depending on how you're interacting with this
thread). :-)

~ Gijs

Benjamin Kerensa

unread,
Apr 21, 2014, 12:27:27 PM4/21/14
to Gijs Kruitbosch, mozilla.dev.planning group, Ehsan Akhgari
On Mon, Apr 21, 2014 at 7:41 AM, Gijs Kruitbosch
<gijskru...@gmail.com> wrote:
> On 21/04/2014 15:24, Michael Connor wrote:
>>
>>
>> On Apr 15, 2014, at 6:59 PM, Gavin Sharp <ga...@gavinsharp.com> wrote:
>>
>>> We need to have
>>> systematic solutions for avoiding expending large amounts of effort
>>> for minimal return, and efficient ways to direct potential
>>> contributors to the right opportunities.

Is there any current policies in place in how you work with
contributors and at what point you stop investing time into a
contributor or group of contributors? Do you have strict criteria you
look for currently such as code examples (Github, Launchpad etc)
before investing time in a contributor?

>>
>>
>> I think this gets held up a lot, but usually in the context of people
>> arguing that we should make calls quicker on people to avoid sinking too
>> much effort into an individual that won’t deliver a good enough return
>> on that investment. My point is that we often can’t know, without
>> investing a significant amount of time, whether someone will be an
>> important contributor over the long run.

I don't think this is something that can be easily solved and I think
all open source projects struggle with investing time in contributors
who just do not work out. On the other hand some other open source
projects do have employees who role is strictly connecting good
contributors and filtering out ones that are not going to work out.

>
>
> I didn't mean to hold this up in terms of "we need quicker calls" but more
> in terms of "how do we communicate such a call". I find it hard to say "no"
> generally, and even harder in these circumstances.
>
> However, I think that the below point is more important:
>
>> Largely, I think we worry a lot about the false positive case (when we
>> invest too much in someone who wasn’t worth it), and don’t spend as much
>> time thinking about the false negative case (when we fail to invest
>> enough time in someone who would be an important contributor). Perhaps
>> more importantly, I think we value the “lost” time for the false
>> positive far more than we value the contributions we’re losing from the
>> false negative case. That’s natural, but I think we can and should do
>> better.
>
>
> Two points here:
> 1) as I pointed out, the false positive and false negative cases are linked
> in that supplies are (relatively) low, and more than a few of the bugs in
> http://www.joshmatthews.net/bugsahoy/?ff=1&unowned=1 were at one point owned
> and went unfixed. All the time in which bugs are assigned to people who
> don't fix them is time they spend making it harder for the "right" people to
> find bugs to work on.

Do we have any tools to check bugs that are assigned for periods of
time and perhaps leave a canned message and unassign the contributor?

>
> 2) How can we reduce the number of false negatives? This is intended as a
> genuine question - I would love to be able to help in that department. Could
> we try to e.g. get team members to identify five mentored bugs a week or
> something? Send swag to people for fixing mentored bugs? Invite them for
> work weeks? Organize "fix mentored bugs" competitions/days? Make it easier
> for people to fix bugs by improving the "effort curve" for contributing?
> Something else? Do we (does Mike) have data on some of this?

I think all of those forms of recognition are good ways to recognize
contributors but I hope that some policy surrounding when each is
offered is developed. For instance I think in the case of inviting
someone to a work week they should be a core contributor to your team
that is contributing at least part time what a paid contributor might.

Also sometimes an e-mail or note is much more rewarding to
contributors then a t-shirt or swag. Just knowing that they are
helping and providing some value is pretty powerful recognition too.

- Benjamin

Gavin Sharp

unread,
Apr 21, 2014, 3:01:58 PM4/21/14
to Michael Connor, Ehsan Akhgari, dev-planning@lists.mozilla.org planning, Gijs Kruitbosch
I worry about the "false positive" case not just because it represents
"wasted effort", but because it's a demotivating experience for the mentor,
which can build on itself and make people less likely to take a chance and
invest in a new contributor in the future. That leads directly to the
"false negative case" that you're describing.

These aren't independent problems - all I'm saying is that we shouldn't
treat them as such, and we need to address them both.

Gavin


On Mon, Apr 21, 2014 at 7:24 AM, Michael Connor <mco...@mozilla.com> wrote:

>
> On Apr 15, 2014, at 6:59 PM, Gavin Sharp <ga...@gavinsharp.com> wrote:
>
> > We need to have
> > systematic solutions for avoiding expending large amounts of effort
> > for minimal return, and efficient ways to direct potential
> > contributors to the right opportunities.
>
> I think this gets held up a lot, but usually in the context of people
> arguing that we should make calls quicker on people to avoid sinking too
> much effort into an individual that won’t deliver a good enough return on
> that investment. My point is that we often can’t know, without investing a
> significant amount of time, whether someone will be an important
> contributor over the long run.
>
> Largely, I think we worry a lot about the false positive case (when we
> invest too much in someone who wasn’t worth it), and don’t spend as much
> time thinking about the false negative case (when we fail to invest enough
> time in someone who would be an important contributor). Perhaps more
> importantly, I think we value the “lost” time for the false positive far
> more than we value the contributions we’re losing from the false negative
> case. That’s natural, but I think we can and should do better.
>
> — Mike

Michael Connor

unread,
Apr 21, 2014, 3:22:58 PM4/21/14
to Gavin Sharp, Ehsan Akhgari, dev-planning@lists.mozilla.org planning, Gijs Kruitbosch
This is an interesting perspective. I’ve mentored plenty of people who didn’t succeed or follow through, and I don’t think I found it demotivating or made me less willing to help others, but that could be an expectations issue. I think I always recognized that I was accepting the false positives to make sure I caught as many good hackers as I could find. It probably helped that my initial job description had community building as 50% of my time, but it was how I was working even when I was still a volunteer.

Describing it as “taking a chance” implies there’s a risk involved for the mentor, but I’m not sure what’s being risked. Is it reputation? Time? Self-confidence (in terms of self-belief in mentoring skills)? Can we help people understand that failures are an expected part of the process, and a necessary evil?

I completely agree that they’re not independent problems. I think it’s a question of “which direction are we happier failing in?” For me, it’s false positives, because I believe that the cost of false positives will be lower than the benefit we’ll get from fewer false negatives. It’d be good to figure out how we measure that to be sure that math holds.

— Mike

Gijs Kruitbosch

unread,
Apr 21, 2014, 3:42:34 PM4/21/14
to
On 21/04/2014 20:22, Michael Connor wrote:
> This is an interesting perspective. I’ve mentored plenty of people who didn’t succeed or follow through, and I don’t think I found it demotivating or made me less willing to help others, but that could be an expectations issue.

Can you give some bugs as examples? How recent was this? I didn't find
anything with the bugsahoy whiteboard tags in bugzilla (for "mconnor")
so I'm wondering if we're just seeing very different types of
contributors now than we have in the past, and so perhaps it's an
experiences rather than expectations issue. :-)

~ Gijs

Mike Hoye

unread,
Apr 21, 2014, 3:54:29 PM4/21/14
to dev-pl...@lists.mozilla.org
On 2014-04-21, 3:22 PM, Michael Connor wrote:
> This is an interesting perspective. I’ve mentored plenty of people who didn’t succeed or follow through, and I don’t think I found it demotivating or made me less willing to help others, but that could be an expectations issue.
For what it's worth, all the developers I've interviewed last year spoke
about the failure of the relationship - not "failure to complete a bug",
mind you, but specifically the failure of the relationship - as being a
major demotivator. So while it's not universally the case, the
demoralizing-false-positive is certainly a real thing. The way I'm
trying to minimize the downside there is by asking people to get into
these relationships cautiously, and to be willing to size up
contributors and redirect enthusiastic but out-of-their-depth people to
something they'll have a good shot at.

We don't have a lot of support for the practice of mentoring in place
right now, which makes a failed mentoring relationship look (and
certainly feel) like a personal failure on the part of the engineer,
rather than the systemic problem that IMO it really represents; we're on
our way there, though, between the mentoring fields coming into Bugzilla
and some better documentation, hopefully this will make for more
functional and easy-to-manage contributor relationships.

- mhoye

Mike Hoye

unread,
Apr 21, 2014, 4:01:00 PM4/21/14
to dev-pl...@lists.mozilla.org
On 2014-04-21, 10:41 AM, Gijs Kruitbosch wrote:
> On 21/04/2014 15:24, Michael Connor wrote:
>> I think this gets held up a lot, but usually in the context of people
>> arguing that we should make calls quicker on people to avoid sinking too
>> much effort into an individual that won’t deliver a good enough return
>> on that investment. My point is that we often can’t know, without
>> investing a significant amount of time, whether someone will be an
>> important contributor over the long run.
>
> I didn't mean to hold this up in terms of "we need quicker calls" but
> more in terms of "how do we communicate such a call". I find it hard
> to say "no" generally, and even harder in these circumstances.

So, I have more to say about this - when I'm back from my trip tomorrow,
one hopes! - but I think that:

- it's pretty clear when somebody is thrashing, rather than making
forward progress, and
- the real challenge is actually making "the call", rather than knowing
that you should.

With regards to what "the call" means, my theory is that having a bunch
of different options for how and where an engineer can send a strugging
contributor close to hand will make this much easier and happier
experience, so I'm working on building those out right now.


- mhoye

Gavin Sharp

unread,
Apr 21, 2014, 4:02:16 PM4/21/14
to Michael Connor, dev-planning@lists.mozilla.org planning
You can't see how mentors might be reluctant to invest time in helping
contributors if a high percentage of the their past efforts have not been
successful in any form? This seems fairly obvious to me. No one is
expecting a 100% "success" rate - but there's no getting around the fact
that a 100% (or 90%) "failure" rate is going to have an impact on your
future efforts.

I fully support encouraging people to invest more in mentoring, and it
makes sense to bias our cost/benefit analysis to the longer term rather
than focusing on short term returns. But you can't boil this down to just
"pick one direction to fail in", because failing too much in the "false
positive" case makes you fail more in the "false negative" case too.

Gavin


On Mon, Apr 21, 2014 at 12:22 PM, Michael Connor <mco...@mozilla.com>wrote:

> This is an interesting perspective. I’ve mentored plenty of people who
> didn’t succeed or follow through, and I don’t think I found it demotivating
> or made me less willing to help others, but that could be an expectations
> issue. I think I always recognized that I was accepting the false
> positives to make sure I caught as many good hackers as I could find. It
> probably helped that my initial job description had community building as
> 50% of my time, but it was how I was working even when I was still a
> volunteer.
>
> Describing it as “taking a chance” implies there’s a risk involved for the
> mentor, but I’m not sure what’s being risked. Is it reputation? Time?
> Self-confidence (in terms of self-belief in mentoring skills)? Can we help
> people understand that failures are an expected part of the process, and a
> necessary evil?
>
> I completely agree that they’re not independent problems. I think it’s a
> question of “which direction are we happier failing in?” For me, it’s
> false positives, because I believe that the cost of false positives will be
> lower than the benefit we’ll get from fewer false negatives. It’d be good
> to figure out how we measure that to be sure that math holds.
>
> — Mike
>
> On Apr 21, 2014, at 3:01 PM, Gavin Sharp <ga...@gavinsharp.com> wrote:
>
> > I worry about the "false positive" case not just because it represents
> "wasted effort", but because it's a demotivating experience for the mentor,
> which can build on itself and make people less likely to take a chance and
> invest in a new contributor in the future. That leads directly to the
> "false negative case" that you're describing.
> >
> > These aren't independent problems - all I'm saying is that we shouldn't
> treat them as such, and we need to address them both.
> >
> > Gavin
> >
> >
> > On Mon, Apr 21, 2014 at 7:24 AM, Michael Connor <mco...@mozilla.com>
> wrote:
> >
> > On Apr 15, 2014, at 6:59 PM, Gavin Sharp <ga...@gavinsharp.com> wrote:
> >
> > > We need to have
> > > systematic solutions for avoiding expending large amounts of effort
> > > for minimal return, and efficient ways to direct potential
> > > contributors to the right opportunities.
> >
> > I think this gets held up a lot, but usually in the context of people
> arguing that we should make calls quicker on people to avoid sinking too
> much effort into an individual that won’t deliver a good enough return on
> that investment. My point is that we often can’t know, without investing a
> significant amount of time, whether someone will be an important
> contributor over the long run.
> >

Mike Hoye

unread,
Apr 21, 2014, 4:13:53 PM4/21/14
to dev-pl...@lists.mozilla.org
On 2014-04-21, 12:27 PM, Benjamin Kerensa wrote:
> Is there any current policies in place in how you work with
> contributors and at what point you stop investing time into a
> contributor or group of contributors? Do you have strict criteria you
> look for currently such as code examples (Github, Launchpad etc)
> before investing time in a contributor?

I wrote a bunch of stuff up here about the mentor's role and expectations:

https://wiki.mozilla.org/Contribute/Coding/Mentoring

They're intended to be helpful guidelines, not policies. No policy or
criteria is going to be able to replace or do better than smart people
who care exercising their best judgement.

> I don't think this is something that can be easily solved and I think
> all open source projects struggle with investing time in contributors
> who just do not work out. On the other hand some other open source
> projects do have employees who role is strictly connecting good
> contributors and filtering out ones that are not going to work out.

Yeah, you can't "solve" people.

> Do we have any tools to check bugs that are assigned for periods of
> time and perhaps leave a canned message and unassign the contributor?

Keeping an eye on this situation is my responsibility; I'm hoping to
automate some of it, but either way.

> I think all of those forms of recognition are good ways to recognize
> contributors but I hope that some policy surrounding when each is
> offered is developed. For instance I think in the case of inviting
> someone to a work week they should be a core contributor to your team
> that is contributing at least part time what a paid contributor might.
>
> Also sometimes an e-mail or note is much more rewarding to
> contributors then a t-shirt or swag. Just knowing that they are
> helping and providing some value is pretty powerful recognition too.
I've outlined a bit of that here:
https://wiki.mozilla.org/Contribute/Coding/Pathways - it's a work in
progress, but it speaks to some of your points.

...but the situation is actually worse than you describe - after a
certain point sending a high-value contributor a t-shirt or some
stickers is kind of insulting; at that point, the rewards that matter
are involvement, recognition and greater responsibility.

- mhoye


Michael Connor

unread,
Apr 21, 2014, 6:42:46 PM4/21/14
to Gijs Kruitbosch, Gavin Sharp, Ehsan Akhgari, dev-planning@lists.mozilla.org planning

On Apr 21, 2014, at 10:41 AM, Gijs Kruitbosch <gijskru...@gmail.com> wrote:

> On 21/04/2014 15:24, Michael Connor wrote:
>>
>> On Apr 15, 2014, at 6:59 PM, Gavin Sharp <ga...@gavinsharp.com> wrote:
>>
>>> We need to have
>>> systematic solutions for avoiding expending large amounts of effort
>>> for minimal return, and efficient ways to direct potential
>>> contributors to the right opportunities.
>>
>> I think this gets held up a lot, but usually in the context of people
>> arguing that we should make calls quicker on people to avoid sinking too
>> much effort into an individual that won’t deliver a good enough return
>> on that investment. My point is that we often can’t know, without
>> investing a significant amount of time, whether someone will be an
>> important contributor over the long run.
>
> I didn't mean to hold this up in terms of "we need quicker calls" but more in terms of "how do we communicate such a call". I find it hard to say "no" generally, and even harder in these circumstances.

These are very difficult conversations to have, for sure. I think a big part of the art of mentoring is about helping people find the right path for them. It’s not just “how do I do this thing?” but “how do I succeed?” If they’re on the wrong path, hopefully you’ve interacted with them enough to help them find a better fit for their skills. If not, and it’s an uncomfortable conversation for you, I’d be happy to engage with individuals to help them find a better place for their energy and passion. I’m sure mhoye and others would also be glad to help out with the hard stuff.

> However, I think that the below point is more important:
>
>> Largely, I think we worry a lot about the false positive case (when we
>> invest too much in someone who wasn’t worth it), and don’t spend as much
>> time thinking about the false negative case (when we fail to invest
>> enough time in someone who would be an important contributor). Perhaps
>> more importantly, I think we value the “lost” time for the false
>> positive far more than we value the contributions we’re losing from the
>> false negative case. That’s natural, but I think we can and should do
>> better.
>
> Two points here:
> 1) as I pointed out, the false positive and false negative cases are linked in that supplies are (relatively) low, and more than a few of the bugs in http://www.joshmatthews.net/bugsahoy/?ff=1&unowned=1 were at one point owned and went unfixed. All the time in which bugs are assigned to people who don't fix them is time they spend making it harder for the "right" people to find bugs to work on.

Are we really in a state where we’re target-constrained? My experience with Mozilla is that there’s effectively infinite possible work. There’s 312 firefox-backlog+ bugs assigned to nobody@, and that number will grow. People should take a bug once they have a patch (though they can comment to say they’re trying to produce a patch.)

All that said, I’d consider it a high quality problem to have so many people wanting to fix bugs that we’re out of useful work to give them.

> 2) How can we reduce the number of false negatives? This is intended as a genuine question - I would love to be able to help in that department. Could we try to e.g. get team members to identify five mentored bugs a week or something? Send swag to people for fixing mentored bugs? Invite them for work weeks? Organize "fix mentored bugs" competitions/days? Make it easier for people to fix bugs by improving the "effort curve" for contributing? Something else? Do we (does Mike) have data on some of this?

Having done this a lot, the following three things, in order, are pretty key to encouraging and including talented people:

* Review turnaround.

If we can tag individuals as “in mentoring” (basically “newbie who’s cleared the #introduction steps”) and then prioritize responses to bugs/patches for that group, we’re winning. Think about how hard it is when reviews get delayed and you have to work on other things, and then come back and fix stuff. Now imagine if the context you’re swapping in is “all of Mozilla tech” and you’ve got a day job or school. Reviews and feedback requests from people in mentoring should trump everyone else short of a chemspill.

I’d suggest “time from taking bug to bug landed and in Nightly” is the key pseudo-metric here. Everyone wants that sense of accomplishment, and we should help them get it quickly.

* Integration into the group.

Get them on IRC, get them interacting with the rest of the team so they feel like they’re part of the project, not outsiders. Have conversations, even somewhat crunchy ones, somewhere they can be a part of it.

* Show them the love.

Swag, work week invites, delivery of tasty things, summits and MozCamps. These are all ways we should people they’re appreciated and important.

These are all hard, and require effort and diligence to get right.

— Mike

Michael Connor

unread,
Apr 21, 2014, 6:45:04 PM4/21/14
to Gijs Kruitbosch, dev-planning@lists.mozilla.org planning
On Apr 21, 2014, at 3:42 PM, Gijs Kruitbosch <gijskru...@gmail.com> wrote:

> On 21/04/2014 20:22, Michael Connor wrote:
>> This is an interesting perspective. I’ve mentored plenty of people who didn’t succeed or follow through, and I don’t think I found it demotivating or made me less willing to help others, but that could be an expectations issue.
>
> Can you give some bugs as examples? How recent was this? I didn't find anything with the bugsahoy whiteboard tags in bugzilla (for "mconnor") so I'm wondering if we're just seeing very different types of contributors now than we have in the past, and so perhaps it's an experiences rather than expectations issue. :-)

I’ve been inactive in mainline dev for a fair while. (I keep filing the points on my head away, but they grow back.) Largely I’m thinking 2005-2010 timeframe. I’m also including things like ‘interns’ in this count. Not every intern is a rock star, even perfect GPA kids from top schools. It happens, and it’s hard to accurately predict.

I’ve seen little evidence that we aren’t attracting the same caliber of people. It’s a bit hard to prove either way, of course.

— Mike

Michael Connor

unread,
Apr 21, 2014, 6:53:51 PM4/21/14
to Benjamin Kerensa, dev-planning@lists.mozilla.org planning, Ehsan Akhgari, Gijs Kruitbosch

On Apr 21, 2014, at 12:27 PM, Benjamin Kerensa <bker...@gmail.com> wrote:

> Is there any current policies in place in how you work with
> contributors and at what point you stop investing time into a
> contributor or group of contributors?

I think this is a judgement call, for the most part. It’s probably worth checking

> Do you have strict criteria you
> look for currently such as code examples (Github, Launchpad etc)
> before investing time in a contributor?

If we do, we’re erring all the way on the side of eliminating false positives. If we didn’t help people without an established track record, we’d be missing a lot of people. Including a number of people who’ve commented on this thread. I’m not convinced that’s a path toward winning.

— Mike

Chris Peterson

unread,
Apr 21, 2014, 8:04:42 PM4/21/14
to
On 4/21/14, 3:45 PM, Michael Connor wrote:
>> >Can you give some bugs as examples? How recent was this? I didn't find anything with the bugsahoy whiteboard tags in bugzilla (for "mconnor") so I'm wondering if we're just seeing very different types of contributors now than we have in the past, and so perhaps it's an experiences rather than expectations issue. :-)
> I’ve been inactive in mainline dev for a fair while. (I keep filing the points on my head away, but they grow back.) Largely I’m thinking 2005-2010 timeframe. I’m also including things like ‘interns’ in this count. Not every intern is a rock star, even perfect GPA kids from top schools. It happens, and it’s hard to accurately predict.

Mike, do we have statistics on how contributors find us? For example,
how many come through Bugs Ahoy, school projects, or fixing a bug that
they reported?


chris

Karl Dubost

unread,
Apr 21, 2014, 8:34:40 PM4/21/14
to Gavin Sharp, dev-planning@lists.mozilla.org planning, Michael Connor

Le 22 avr. 2014 à 05:02, Gavin Sharp <ga...@gavinsharp.com> a écrit :
> expecting a 100% "success" rate - but there's no getting around the fact that a 100% (or 90%) "failure" rate is going to have an impact on your future efforts.

That would mean you have been able in advance to define what is "success" and what is "failure". My experience (not at Mozilla), but at W3C in terms of contributors to the W3C validators, spec reviews, etc. is that the return on investment is far to be even and success is a very difficult thing to assess.

* There are people who have completed the one task that they wanted to do and will never come back. It was successful because it was their initial goal.

* There are people who understood, and will not participate, BUT, WHEN the time permits in the future, will contribute actively.

* There are people where the mentoring didn't lead to direct contributions but to them being strong advocates for the project leading other people to come. They came to learn how it was happening and in return encourage others to do.

I think in all these discussions we are having we forget about our own attitudes with regards to other projects in life, sports club, etc. We met people who gave us a bit of our time to understand an area, … sometimes we pushed further, sometimes we gave up.
It. is. ok.
The people who mentored us, shared, etc. did it out of love for what they are doing, not for productivity marks ;)

Now it is entirely possible (and acceptable) that someone doesn't want to do that :) No issue.



--
Karl Dubost, Mozilla
http://www.la-grange.net/karl/moz

Gavin Sharp

unread,
Apr 21, 2014, 9:27:05 PM4/21/14
to Karl Dubost, dev-planning@lists.mozilla.org planning
On Mon, Apr 21, 2014 at 5:34 PM, Karl Dubost <kdu...@mozilla.com> wrote:

>
> Le 22 avr. 2014 à 05:02, Gavin Sharp <ga...@gavinsharp.com> a écrit :
> > expecting a 100% "success" rate - but there's no getting around the fact
> that a 100% (or 90%) "failure" rate is going to have an impact on your
> future efforts.
>
> That would mean you have been able in advance to define what is "success"
> and what is "failure".
>

No, I think the point I'm making is independent of whatever definitions you
use for those words (as long as you recognize that both are possible
outcomes). I agree with you that "success" can mean many different things,
and we shouldn't define it too narrowly.

Gavin

Ehsan Akhgari

unread,
Apr 21, 2014, 10:55:51 PM4/21/14
to Gavin Sharp, Michael Connor, dev-planning@lists.mozilla.org planning
On 2014-04-21, 4:02 PM, Gavin Sharp wrote:
> You can't see how mentors might be reluctant to invest time in helping
> contributors if a high percentage of the their past efforts have not been
> successful in any form? This seems fairly obvious to me. No one is
> expecting a 100% "success" rate - but there's no getting around the fact
> that a 100% (or 90%) "failure" rate is going to have an impact on your
> future efforts.

Yeah, I see this problem. But can't we try to address this by educating
the mentors about this?

FWIW I'd be curious to see if we can gather data about the experiences
of the people who have mentored new contributors in the past. That way
we can approach this problem with some concrete data.

Cheers,
Ehsan

Ehsan Akhgari

unread,
Apr 21, 2014, 11:09:04 PM4/21/14
to Michael Connor, Gijs Kruitbosch, Gavin Sharp, dev-planning@lists.mozilla.org planning
On 2014-04-21, 6:42 PM, Michael Connor wrote:
>> 2) How can we reduce the number of false negatives? This is intended as a genuine question - I would love to be able to help in that department. Could we try to e.g. get team members to identify five mentored bugs a week or something? Send swag to people for fixing mentored bugs? Invite them for work weeks? Organize "fix mentored bugs" competitions/days? Make it easier for people to fix bugs by improving the "effort curve" for contributing? Something else? Do we (does Mike) have data on some of this?
>
> Having done this a lot, the following three things, in order, are pretty key to encouraging and including talented people:
>
> * Review turnaround.
>
> If we can tag individuals as “in mentoring” (basically “newbie who’s cleared the #introduction steps”) and then prioritize responses to bugs/patches for that group, we’re winning. Think about how hard it is when reviews get delayed and you have to work on other things, and then come back and fix stuff. Now imagine if the context you’re swapping in is “all of Mozilla tech” and you’ve got a day job or school. Reviews and feedback requests from people in mentoring should trump everyone else short of a chemspill.

This. I'd actually go as far as saying that as far as a potential
future contributor is concerned, the review turn-around time is a sign
of respect for their contribution from the project's part.

As a personal rule, if I think that I won't be able to handle my review
in a given day, I always prioritize review requests from people I don't
know. Many of them turn out to be new contributors (whether paid or not.)

> I’d suggest “time from taking bug to bug landed and in Nightly” is the key pseudo-metric here. Everyone wants that sense of accomplishment, and we should help them get it quickly.

Yes, delayed gratification rarely wins.

> * Integration into the group.
>
> Get them on IRC, get them interacting with the rest of the team so they feel like they’re part of the project, not outsiders. Have conversations, even somewhat crunchy ones, somewhere they can be a part of it.

I'd like to add a related entry:

* Help them make progress.

Often after I help someone lands his first patch at Mozilla, I politely
indicate to them that I'd be available to help the take on a second bug
if they're interested. If they say they are, I'll usually suggest some
other bugs to them to look at, and also try to make it clear that if
they have their own idea of what they would want to work on next that's
OK too.

> * Show them the love.
>
> Swag, work week invites, delivery of tasty things, summits and MozCamps. These are all ways we should people they’re appreciated and important.
>
> These are all hard, and require effort and diligence to get right.

There's also some psychology involved, watch this video for example
<https://www.youtube.com/watch?v=u6XAPnuFjJc> which summarizes a lot of
good points about how to motivate people.

Cheers,
Ehsan


Mike Hoye

unread,
Apr 22, 2014, 8:19:48 AM4/22/14
to dev-pl...@lists.mozilla.org
On 2014-04-21, 11:09 PM, Ehsan Akhgari wrote:
>
> This. I'd actually go as far as saying that as far as a potential
> future contributor is concerned, the review turn-around time is a sign
> of respect for their contribution from the project's part.
>
> As a personal rule, if I think that I won't be able to handle my
> review in a given day, I always prioritize review requests from people
> I don't know. Many of them turn out to be new contributors (whether
> paid or not.)
>

This is a measurably important thing; we learned last year at MSR2013
that speed of response time to a patch is strongly correlated to
retention. If we turn around a review in the first day, we're quite
likely to see a second patch from that contributor. If we wait a week,
that contributor is very likely to move on.


- mhoye

Michael Connor

unread,
Apr 22, 2014, 10:59:29 AM4/22/14
to Gavin Sharp, dev-planning@lists.mozilla.org planning
On Apr 21, 2014, at 4:02 PM, Gavin Sharp <ga...@gavinsharp.com> wrote:

> You can't see how mentors might be reluctant to invest time in helping contributors if a high percentage of the their past efforts have not been successful in any form? This seems fairly obvious to me. No one is expecting a 100% "success" rate - but there's no getting around the fact that a 100% (or 90%) "failure" rate is going to have an impact on your future efforts.

If we're seeing a 100% failure rate (or even 90%), we should be doing some sort of group post-mortem to understand why it’s that bad. That’s not a number I’ve heard previously. If it’s just some people, that would suggest the mentors need help and coaching. If it’s everyone, we’ve probably got deeper issues than how we identify and classify good bugs.

> I fully support encouraging people to invest more in mentoring, and it makes sense to bias our cost/benefit analysis to the longer term rather than focusing on short term returns. But you can't boil this down to just "pick one direction to fail in", because failing too much in the "false positive" case makes you fail more in the "false negative" case too.

It’d be absurd to take this to extremes, and that’s not my argument. My point is that in deciding how far we turn the dial on “effort we expend to get a contributor up to speed” we’re either going to over-invest in people who aren’t worth it, or under-invest in people who are worth it. Do we want more useful contributors, at the expense of more wasted time, or should we run leaner and only invest time in the obviously awesome people? How do we measure and optimize our returns here?

— Mike

Mike Hoye

unread,
Apr 22, 2014, 11:01:16 AM4/22/14
to dev-pl...@lists.mozilla.org
On 2014-04-21, 10:55 PM, Ehsan Akhgari wrote:
> On 2014-04-21, 4:02 PM, Gavin Sharp wrote:
>> You can't see how mentors might be reluctant to invest time in helping
>> contributors if a high percentage of the their past efforts have not
>> been
>> successful in any form? This seems fairly obvious to me. No one is
>> expecting a 100% "success" rate - but there's no getting around the fact
>> that a 100% (or 90%) "failure" rate is going to have an impact on your
>> future efforts.
>
> Yeah, I see this problem. But can't we try to address this by
> educating the mentors about this?
>
> FWIW I'd be curious to see if we can gather data about the experiences
> of the people who have mentored new contributors in the past. That
> way we can approach this problem with some concrete data.

I interviewed a dozen or so engineers last year; not a huge sample, but
I for the few common points among them I think reasonably representative.

The highlights - lowlights, really? - of that process were that everyone
agrees that helping new contributors grow is important and wants to do
more, but developers generally feel equally bad about three things:
- cutting off aspiring contributors when it becomes clear they're
thrashing/out of their depth/getting nowhere etc.
- having their mentoring relationship drag on unproductively at
painful length, and
- not having a clear sense of where mentoring falls onto their list
of priorities and quarterly goals.

Everyone I spoke to made some variation of these three points as being
the most difficult things about mentoring contributors, so my hope for
addressing them is:

- To make it clear to new contributors, per-bug, what Mozilla's
expectations are as far as readiness and experience are, and what we
think will make for a successful contribution
- To set out some guidelines for mentors (check in every week, show some
prior work, etc...) about how to make that relationship work and how to
positively direct a contributor towards a better fit when it's not
working, and
- To advocate that engineers be given quarterly goals around engagement
and the time to pursue them successfully.

That last thing I'm beating the drums for is basically "one mentored bug
in-progress per product cycle per engineer", which I think given some
better context - clear next steps for both parties, good alternative
plans and a clear sense that even with the best resources and intentions
sometimes it won't work and that's fine, it should be a better and more
productive experience for all concerned.


- mhoye

Boris Zbarsky

unread,
Apr 22, 2014, 11:51:26 AM4/22/14
to
On 4/22/14, 10:59 AM, Michael Connor wrote:
> If we're seeing a 100% failure rate (or even 90%), we should be doing some sort of group post-mortem to understand why it’s that bad.

Anecdote, not data: a lot of people I've dealt with recently seem to be
showing up trying to contribute because they're university students with
a "contribute a patch to an open-source project" homework assignment.
This is becoming reasonably common in some countries (just like the
"report a bug in bugzilla" homework assignment that causes useless bug
reports).

This tends to not work all that great even for the first patch, and
tends to have near-0 post-first-patch retention....

-Boris

Chris Hofmann

unread,
Apr 22, 2014, 12:15:12 PM4/22/14
to Boris Zbarsky, dev-pl...@lists.mozilla.org
It makes sense to have a different path and different set of success
metrics for these kind of contributors.

First thing might be to try and trace back to the institutions that are
encouraging/requiring this kind of work. Having a list of contacts
there could help to improve the communication and make these limited
contributions more successful.

Building a set of Teaching Assistants within those organizations or
within Mozilla could keep the regular contributors/mentors building new
contributors that plan to stick around for more than just one "assignment."

Again, its my experience that the ratio's look something like this.

Talk to 1000 people. Nine or Ten show some interest in contributing.
One or two actually contribute something. After awhile this feels like
100% failure rate, but those 1 or 2 are critical to building and
sustaining the project going forward. The lessons here are about
setting the right expectations. Mentoring and expanding the project
with more contributors is really hard work. That doesn't mean we should
not do it.

-chofmann

Valentin Gosu

unread,
Apr 22, 2014, 1:22:50 PM4/22/14
to chof...@mozilla.org, Boris Zbarsky, dev-pl...@lists.mozilla.org
On 22 April 2014 19:15, Chris Hofmann <chof...@mozilla.com> wrote:

> On 4/22/14 8:51 AM, Boris Zbarsky wrote:
>
> It makes sense to have a different path and different set of success
> metrics for these kind of contributors.
>
> First thing might be to try and trace back to the institutions that are
> encouraging/requiring this kind of work. Having a list of contacts there
> could help to improve the communication and make these limited
> contributions more successful.
>
> Building a set of Teaching Assistants within those organizations or within
> Mozilla could keep the regular contributors/mentors building new
> contributors that plan to stick around for more than just one "assignment."
>
> Again, its my experience that the ratio's look something like this.
>
> Talk to 1000 people. Nine or Ten show some interest in contributing.
> One or two actually contribute something. After awhile this feels like
> 100% failure rate, but those 1 or 2 are critical to building and sustaining
> the project going forward. The lessons here are about setting the right
> expectations. Mentoring and expanding the project with more contributors
> is really hard work. That doesn't mean we should not do it.
>

As someone who suggested open-source contributions as bonus assignments to
students, I can confirm that doing so in bulk resulted in a very low
percentage of a second patch (less than 5%). However, with two of the ones
that contributed one initial patch, I was able to get in touch with, and
mentor for an entire summer. This resulted in several good and useful
patches, increased involvement, and one intern for the upcoming summer.

Some of the things I noted:
- contributor involvement is directly proportionate with the time you put
in mentoring
- reviews that take more than 3 days are a huge turn-off (especially for
new contributors, since they're doing one bug at a time)
- in-person meetings are really productive, and video meetings are better
that IRC
- new contributors aren't used to asking questions. Very often they get
stuck on something, and then they go away.
- challenging bugs are usually more engaging than simple bugs, if there is
good mentor to point them in the right direction.

While I agree that 1 or 2 people out of 10 who show an interest is fine, I
think we could do better. If we could set up a small group (reps maybe?)
tasked with reaching out to new contributors and helping them to get
started, I think our success rate would be much higher.

Mike Hoye

unread,
Apr 22, 2014, 2:16:19 PM4/22/14
to dev-pl...@lists.mozilla.org
On 2014-04-22, 12:15 PM, Chris Hofmann wrote:
> On 4/22/14 8:51 AM, Boris Zbarsky wrote:
>>
>> Anecdote, not data: a lot of people I've dealt with recently seem to
>> be showing up trying to contribute because they're university
>> students with a "contribute a patch to an open-source project"
>> homework assignment. This is becoming reasonably common in some
>> countries (just like the "report a bug in bugzilla" homework
>> assignment that causes useless bug reports).
>>
>> This tends to not work all that great even for the first patch, and
>> tends to have near-0 post-first-patch retention....
>>
>> -Boris
>
> It makes sense to have a different path and different set of success
> metrics for these kind of contributors.

We're not really equipped to treat these contributors differently at
this point, and I don't think that it would be a positive step if we
were. There's not a lot we can do about ill-considered curriculum
decisions that want to use us as free instructional labor by the time
the curriculum's been decided. What we really need is a way to be in the
room when those decisions are getting made.

I think that we need to engage with the institutions pursuing these
initiatives, rather than the individuals doing the filing. Right now we
don't really have a setup that's compatible with student projects -
single-contributor focus and two- or six-week cycles vs. small teams and
term-based cycles - and making new Dave Humphreys doesn't seem to be in
the cards.

If a prof or institution is ready to collaborate with us, we should be
able - and in many cases already can - connect them with a reasonable
set of bugs, projects or goals, maybe even regionally relevant goals
with the Reps involvement. But I think that this is the responsibility
of the larger Engagement team, but that beyond having a short list of
nice-to-have speculative-but-not-priority features, I don't think core
engineering specifically needs to invest a lot of effort into to this.


- mhoye

Gavin Sharp

unread,
Apr 22, 2014, 2:33:38 PM4/22/14
to Mike Hoye, dev. planning
On Tue, Apr 22, 2014 at 8:01 AM, Mike Hoye <mh...@mozilla.com> wrote:
> Everyone I spoke to made some variation of these three points as being the
> most difficult things about mentoring contributors, so my hope for
> addressing them is:
>
> - To make it clear to new contributors, per-bug, what Mozilla's expectations
> are as far as readiness and experience are, and what we think will make for
> a successful contribution
> - To set out some guidelines for mentors (check in every week, show some
> prior work, etc...) about how to make that relationship work and how to
> positively direct a contributor towards a better fit when it's not working,
> and
> - To advocate that engineers be given quarterly goals around engagement and
> the time to pursue them successfully.

This sounds great.

Gavin

Brandon Benvie

unread,
Apr 22, 2014, 3:15:28 PM4/22/14
to Mike Hoye, dev-pl...@lists.mozilla.org
Sounds like we need a Mozilla school. Then we'd be in the room when the decisions were made.

> On Apr 22, 2014, at 11:17 AM, Mike Hoye <mh...@mozilla.com> wrote:
>
>> On 2014-04-22, 12:15 PM, Chris Hofmann wrote:
>>> On 4/22/14 8:51 AM, Boris Zbarsky wrote:
>>>
>>> Anecdote, not data: a lot of people I've dealt with recently seem to be showing up trying to contribute because they're university students with a "contribute a patch to an open-source project" homework assignment. This is becoming reasonably common in some countries (just like the "report a bug in bugzilla" homework assignment that causes useless bug reports).
>>>
>>> This tends to not work all that great even for the first patch, and tends to have near-0 post-first-patch retention....
>>>
>>> -Boris
>>
>> It makes sense to have a different path and different set of success metrics for these kind of contributors.
>
> We're not really equipped to treat these contributors differently at this point, and I don't think that it would be a positive step if we were. There's not a lot we can do about ill-considered curriculum decisions that want to use us as free instructional labor by the time the curriculum's been decided. What we really need is a way to be in the room when those decisions are getting made.
>
> I think that we need to engage with the institutions pursuing these initiatives, rather than the individuals doing the filing. Right now we don't really have a setup that's compatible with student projects - single-contributor focus and two- or six-week cycles vs. small teams and term-based cycles - and making new Dave Humphreys doesn't seem to be in the cards.
>
> If a prof or institution is ready to collaborate with us, we should be able - and in many cases already can - connect them with a reasonable set of bugs, projects or goals, maybe even regionally relevant goals with the Reps involvement. But I think that this is the responsibility of the larger Engagement team, but that beyond having a short list of nice-to-have speculative-but-not-priority features, I don't think core engineering specifically needs to invest a lot of effort into to this.
>
>
> - mhoye
> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning

Naoki Hirata

unread,
Apr 22, 2014, 3:20:16 PM4/22/14
to Brandon Benvie, dev-pl...@lists.mozilla.org, Mike Hoye
That sounds like a really cool notion. Should we explore it?

On Apr 22, 2014, at 12:15 PM, Brandon Benvie <bbe...@mozilla.com> wrote:

> Sounds like we need a Mozilla school. Then we'd be in the room when the decisions were made.
>
>> On Apr 22, 2014, at 11:17 AM, Mike Hoye <mh...@mozilla.com> wrote:
>>
>>> On 2014-04-22, 12:15 PM, Chris Hofmann wrote:
>>>> On 4/22/14 8:51 AM, Boris Zbarsky wrote:
>>>>
>>>> Anecdote, not data: a lot of people I've dealt with recently seem to be showing up trying to contribute because they're university students with a "contribute a patch to an open-source project" homework assignment. This is becoming reasonably common in some countries (just like the "report a bug in bugzilla" homework assignment that causes useless bug reports).
>>>>
>>>> This tends to not work all that great even for the first patch, and tends to have near-0 post-first-patch retention....
>>>>
>>>> -Boris
>>>

Mike Hoye

unread,
Apr 22, 2014, 3:35:40 PM4/22/14
to dev-pl...@lists.mozilla.org
On 2014-04-22, 3:20 PM, Naoki Hirata wrote:
> That sounds like a really cool notion. Should we explore it?
>

Right now - right this week, in fact! - the engagement team is
developing a set of Mozcamp sessions to teach mentoring and
domain-specific advocacy and recruitment skills, with the goal of
levelling up contributors to mentors and evangelists.

We'll have more to report on that in a few days, so stay tuned!

- mhoye

Chris Hofmann

unread,
Apr 22, 2014, 4:01:07 PM4/22/14
to dev-pl...@lists.mozilla.org

>> On Apr 22, 2014, at 11:17 AM, Mike Hoye <mh...@mozilla.com> wrote:
>>
>>> On 2014-04-22, 12:15 PM, Chris Hofmann wrote:
>>>> On 4/22/14 8:51 AM, Boris Zbarsky wrote:
>>>>
>>>> Anecdote, not data: a lot of people I've dealt with recently seem to be showing up trying to contribute because they're university students with a "contribute a patch to an open-source project" homework assignment. This is becoming reasonably common in some countries (just like the "report a bug in bugzilla" homework assignment that causes useless bug reports).
>>>>
>>>> This tends to not work all that great even for the first patch, and tends to have near-0 post-first-patch retention....
>>>>
>>>> -Boris
>>> It makes sense to have a different path and different set of success metrics for these kind of contributors.
>> We're not really equipped to treat these contributors differently at this point, and I don't think that it would be a positive step if we were.
If we don't figure out a plan for these kind of contributions we are
setting ourselves up for one class of mentoring failures.

Mentor spends lots of time and effort helping a student along, then
student disappears after one bug. Mentor seeks ways to avoid that
including not doing the mentoring or fast reviews....

>> There's not a lot we can do about ill-considered curriculum decisions that want to use us as free instructional labor by the time the curriculum's been decided.

Yep, your right. That's one of the wonders of open source/free software
projects...

So we should look for easy ways to get in tune with what's going on.
Seems like asking a few new contributors where they are coming from,
what's there motivation, and if student what
institution-class-instructor sent them our way up front, instead of
after is an easy way to get a ahead of this a bit.

>> What we really need is a way to be in the room when those decisions are getting made.
or just try and circulate some information (wiki page) about how to make
these kinds of contributions and participation more successful. Get
the instructors/professors to share their thoughts there...
>> I think that we need to engage with the institutions pursuing these initiatives, rather than the individuals doing the filing. Right now we don't really have a setup that's compatible with student projects - single-contributor focus and two- or six-week cycles vs. small teams and term-based cycles - and making new Dave Humphreys doesn't seem to be in the cards.

agree that that "high touch" kind of effort is not in the cards. I'm
suggesting "lower touch" ways of communicating and guiding these kinds
of efforts.
>> If a prof or institution is ready to collaborate with us, we should be able - and in many cases already can - connect them with a reasonable set of bugs, projects or goals, maybe even regionally relevant goals with the Reps involvement.
but that's not happening. do reps really know what to tell the profs?
could they know and understand what to tell them?

-chofmann

Mike Hoye

unread,
Apr 22, 2014, 4:57:03 PM4/22/14
to dev-pl...@lists.mozilla.org
On 2014-04-22, 4:01 PM, Chris Hofmann wrote:
>
>>> On Apr 22, 2014, at 11:17 AM, Mike Hoye <mh...@mozilla.com> wrote:
>>> We're not really equipped to treat these contributors differently at
>>> this point, and I don't think that it would be a positive step if we
>>> were.
> If we don't figure out a plan for these kind of contributions we are
> setting ourselves up for one class of mentoring failures.
>
> Mentor spends lots of time and effort helping a student along, then
> student disappears after one bug. Mentor seeks ways to avoid that
> including not doing the mentoring or fast reviews....
A contributor that completes one bug without coming back is not the best
possible outcome, but it is certainly not a failure.

But I stand by my claim: by the time we discern (if we manage to do that
at all) that this is a thing, it's too late to change the curriculum or
the assignments.

We can't make this engineering's problem; kids who've been set up to
fail by an instructor who didn't realize what they were asking for are
out of any scope we want to be a part of, and trying to address that
problem will be just heartbreak like turtles, all the way down.

The only current-term reaction that makes sense is what we're already
doing for individuals. The long game after that is for the engagement
team to reach out to the institution to try to help create an
achievable, relevant curriculum for the next wave of students.


- mhoye

Florian Quèze

unread,
Apr 23, 2014, 9:48:26 AM4/23/14
to Mike Hoye, dev. planning
On Sun, Mar 30, 2014 at 10:49 PM, Mike Hoye <mh...@mozilla.com> wrote:

> - The Mentoring doc spells out what makes a good first bug and a
> well-functioning mentoring relationship.

I wrote down my thoughts about mentoring Google Summer of Code students:
http://blog.queze.net/post/2014/04/22/Thoughts-about-mentoring-Summer-of-Code-students

I think GSoC is an excellent opportunity to get new long term
contributors, but it requires good mentoring.

One of the key difference between what's already been discussed here
and the GSoC situation is that you don't have to worry about whether
the student will submit a second patch after the first one is
finished; but you instead need to motivate the student to stay around
and continue to contribute after the final payment is received.

Florian

--
Florian Quèze

Mike Hoye

unread,
Apr 23, 2014, 10:36:37 AM4/23/14
to dev-pl...@lists.mozilla.org
Not yet. We've got a project underway called Baloo:

https://wiki.mozilla.org/Baloo

... with that as its goal, and we're connecting as many sites as we can
into Google Analytics, but we haven't tied the whole picture together yet.

I actually wrote "tied the whole picture together" there, like that's a
coherent sentence that will evoke a clear, comprehensible picture in
your imagination. Brain, what good are you, why are you even here.

Our data is incomplete but growing fast, and I'll have more to say about
that soon.


- mhoye

Mike Hoye

unread,
Apr 23, 2014, 11:59:27 AM4/23/14
to dev-pl...@lists.mozilla.org
On 2014-04-21, 6:53 PM, Michael Connor wrote:
> On Apr 21, 2014, at 12:27 PM, Benjamin Kerensa <bker...@gmail.com> wrote:
>
>> Is there any current policies in place in how you work with
>> contributors and at what point you stop investing time into a
>> contributor or group of contributors?
> I think this is a judgement call, for the most part. It’s probably worth checking
I've written up some best-practices - some even data-driven! - here:

https://wiki.mozilla.org/Contribute/Coding/Mentoring#Good_Mentored_Bugs

But in short, contributors:
- Must have their dev environment set up and building successfully
before they take on a mentored bug,
- Might be asked to show some prior work or a tentative first run at a
patch before they can take the bug,
- Should check in once a week, so everyone can see things are
progressing, and
- Must respect their mentor's guidance with respect to patch
submission, review and direction towards a better-fitting opportunity if
progress stalls.

Mentors, in turn:
- Should be judicious about who you agree to mentor,
- Must get back to mentorees' inquiries within 24 hours, and
- Should regularly evaluate if progress is being made, and redirect
the contributor to another opportunity if they've stalled out, and
- Should promptly deassign the bug as soon as they feel the
contributor has lost interest/disappeared.

"Radio silent for two weeks" is a pretty reasonable threshold for
pulling the trigger on that last part, I think, but as always these are
guidelines, not policies. No rule will ever be more important than the
mentor's best judgement.

- mhoye
0 new messages