Following up - Entrypoint bugs and good mentoring experiences.

[Michael Hoye]

Hi, folks. Just a head's up, some of this wall of text is engineering-specific, but I hope you'll wade through it. There's something here for everyone.

So, following up: I've interviewed a bunch of people around Mozilla to try and answer some questions I've had about bugs, what makes a good first bug, mentored bug and mentored bug experience, that sort of thing, and here we are.

My main hope in sending this is this: I don't want to radically change the way people approach mentored or student bugs, but I'd like us to start talking about this stuff using the same words.

Right now, there actually isn't much disagreement about what these categories should be, but I've found that different people use different words to hide their vigorous agreement from each other, often with much success. Possibly worse, blurring the terminology has meant we all have very different expectations about what a successful experience with new-contributor bugs looks like.

I'm going to propose a few definitions, in the hope that they stick. We've got four terms to describe, for lack of a better term, new-contributor-entry-point bugs.

Here's what they are, and what differentiates them:

Good First Bug - A good first bug is the very simplest of starter bugs, a problem for which (at least as far as Engineering goes) we expect contributors to spend more time actually spinning up their build environment than finding and fixing the bug. Key points that many people made are that bugs of this type have two important qualities; they're highly localized - ideally needing changes to as little as one file - and they aren't on anyone's critical path to anywhere. A contributor should be able to provide a fix or patch for a good first bug with a minimum amount of assistance or intervention from an employee, and there aren't any scheduling constraints on the ability of a contributor to pursue a "good first bug".

The Good Student Bug - a "good student bug" has a lot of the same qualities - narrow in scope, etc - with the critical distinction that students are on deadline; they're going to hit those bugs in waves every four months, and their grades depend on their ability to deliver some sort of reasonably complete, gradeable result in the next three months, probably while doing three or four other classes.

Student Project - A good student project looks very similar, but they happen over a longer timescale - 2+ terms - involve somewhat more ambitious deliverables, as much as an entire feature or addon, and coordinating with the assigning prof more closely.

Mentored Bugs - For mentored bugs a contributor is expected to have their build environment already spun up, perhaps obviously, and even though it needs to be a real thing that will really ship, mentored bugs should likewise not be on anyone's critical path. The difference between a good first bug and a good mentored bug is, as far as I can tell, twofold: first, a Mozilla engineer is willing to commit time to mentoring the developer, and second, the contributor has demonstrated some - perhaps not much, but a promising "some" - domain expertise in the area around the bug, and a willingness to take feedback and grow. Again, time constraints don't really apply to this kind of bug.

Oddly, one big challenge we have in identifying these bugs is not actually fixing them. Most of the time the amount of work it takes to evaluate a bug for assignment, especially for the student bugs, is _very_ close to the amount of work it would take for an experienced developer to actually fix the bug. I'll come back to this in a second.

The most consistent reply I've heard, when I asked people what makes for a good mentoring _experience_, as opposed to a good mentored bug (again: this is mostly focused on core engineering) is:

- regular communication,
- regular small course-corrections and mentoring sessions.
- regular, visible forward progress.
- … and finally (and often emphatically) that the person they're dealing with is not so green that they need to be walked through every step of getting anything done. This has been described by many people as "very frustrating" and "a waste of time".

Did I say I that I didn't want to change how people treated this stuff? Well, not radically change, at least.

The biggest disappointment across all the entry-point bugs is when one part of that process just… disappears. Sometimes, rarely but sometimes, that's us; a Mozilla employee leaves for a holiday or parental leave or some other company and it doesn't get handed off, sometimes it gets misflagged (flagged for review without calling out a reviewer, that gets mentioned a _lot_) and sometimes a bug just falls through the cracks of an overburdened inbox.

Sometimes, the contributor goes dark; stops responding to hails, the bug goes moribund because they, for whatever reason, doesn't continue. We don't have a lot of insight into why that is yet - maybe life just happens to them and they need to step away, maybe they've bitten off more than they can chew, maybe we could be better about clarifying what it's going to take to ship a patch… lots of maybes. I'm working on them.

I think there are two ideas that I'd like to propose here, that sort of involve automation and sort of involve managing expectations and might make our lives easier.

The first is: I think we need a time limit on the activity in entry-point bugs, whatever they are - two weeks seems like a sane guess, but I'm not sure - after which the mentor who's supervising that work can say "This isn't working, we're throwing this one back in the water", and make the bug available for other contributors again. No hard feelings, burdens on your conscience or career repercussions, just throw it back and move on. We can automate that (or at least, we will be able to soon) but if we get some consensus on this, we should make our expectations of contributors explicit.

For the second: many of you are familiar with the idea of "minimum viable product" question, "what's the smallest, simplest thing we can get out the door that will make somebody's life better." The thing I'd like you (you, mozilla-engineering, sure, but you "anyone who wants to work with contributors on your cool stuff at Mozilla" too) to give some thought to this: what constitutes the minimum viable contributor, to your corner of the Mozilla project?

For core engineering, I think the answer might be "Can build Firefox on their machine, can comment on a bug, and can find us on IRC or email to talk about it." Other teams might have extremely different answers - Localization might say, "can speak two languages and edit a wiki", as one example I've just made up. There doesn't seem to be any correlation between how complicated a full-time job at Mozilla is and how hard it is for a contributor to add value to it; as one example, Release Engineering has told me straight up that the atomic value of useful contributions to them is somebody who can "stare at a webpage, see colours, notice a trend and tell us."

The reason I'd like everyone to consider this is so that we start managing expectations, employees and contributors alike, very early in the process, and take incoming contributors who don't already have their hearts set on what they'd like to do and give them a sense of where they'd be a good fit, what it takes in that area to make a valuable contribution to Mozilla. And so they can have some clarity about what they can expect from us in terms of effort and recognition in return.



There's more to say here - Badges are going to be a thing on Mozillians, there's more to be said about how we can recognize contributors' efforts and streamline the involvement process, what we can do about contributor retention rates, these are all important questions - but this has gone on long enough. I'd like to figure out how to recognize and reward great mentors, as well, because that's a real thing we should really be doing.

For now:

- If you find something that's going to be super-easy to fix, maybe try and stop yourself from fixing it. Just flag it, and give somebody new someplace to cut their teeth.
- Please get back to your mentored bugs quickly, for real.
- If it's not working out after a good effort, throw them back.

and:

- Are these good buckets for entry-point bugs? That seems to be the consensus, but if you disagree, I'm all ears.
- Tell me what makes an MVC in your area of the project.

I'd love your feedback.

Thanks for your time, and thanks to everyone who's taken the time to answer my questions about this over the last two weeks.

- mhoye

[David Humphrey]

Thanks for taking this on, Mike. There's lots of great stuff in this
email, and I'm still processing it. A couple of things that I thought
about as I read it.

First, one area that I don't think we're properly exploiting is all the
million ways one can contribute to Mozilla outside Bugzilla. Many of
the efforts you describe below have been facilitated by keywords in
BMO. I think that's been really useful (I know I've used it to get
hundreds of students involved). However, Mozilla isn't only working in
Bugzilla now. As David Ascher said to me yesterday, "the web community
is all on github." Go take a look at https://github.com/mozilla/ --533
repos! Obviously not all of them are the right place to put new people,
but if even half are, it's a huge opportunity to also get a system that
surfaces similar opportunities. Mozilla is massive. I think we need
something like what jdm did with http://whatcanidoformozilla.org/ that
surfaces opportunities separate to any particular tool, repo, or bug
tracker.

Second, I think it's important to figure out how to do mentoring across
sub-communities in addition to individually. What I mean by this is
that I have often found that trying to get a student or new contributor
connected with a person is almost certainly going to fail, and for
reasons you've outlined (holidays, leave, busy with a release, ...).
However, where I've had the most success it's been with a small group of
people who can spell each other off. For example, there are
half-a-dozen people in #content that I know will help one of my students
at any given time. However, if I tried to pin one of them down, and
formalize it, I don't think it would work, or at least wouldn't work at
scale. Finding ways for an area of Mozilla to mentor vs. a person is key.

Third, I want to give some credit to #introduction, which is really
kicking ass on irc. If you're not there, and you want to help people,
I'd suggest you join. Just last night I watched as one of my students
built up the nerve to ask a question, sure he'd be told he was an
idiot. Not 30 seconds later khuey had an answer for him, and the
student left with a deeper understanding of his problem. "Wow, I'm
amazed at how helpful they are in there," he told me in class today.
Where else can a new dev go and rub shoulders with such talented
people? It's amazing.

Finally, I really like what you said about "Good First Bug," and how it
should be more about learning the process and setting up your
environment than actually solving problems. Yesterday we had two new
developers start working on the Webmaker project with bugs to fix
spelling mistakes. In the process they got to learn our review process,
where code lives, who to talk to, etc. These bugs want to be trivial,
and a way to have a first success--if you will put in the time, clone
our repo, build the code, and modify one line, you're one of us. That's
the message with these bugs.

Looking forward to seeing where you'll take all this. It's great to see
focus happening here.

Dave

> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning

[Justin Dolske]

On 3/7/13 2:02 PM, Michael Hoye wrote:

> Hi, folks. Just a head's up, some of this wall of text is engineering-specific, but I hope you'll wade through it. There's something here for everyone.

Indeed there is -- lots of great stuff!

> The Good Student Bug - [...]
> Student Project - [...]

I guess the implication here is that we would have a set of bugs aimed
at students? Sounds like a fine idea, I've just not heard of it before.

> The first is: I think we need a time limit on the activity in entry-point bugs, whatever they are - two weeks seems like a sane guess, but I'm not sure - after which the mentor who's supervising that work can say "This isn't working, we're throwing this one back in the water", and make the bug available for other contributors again. No hard feelings, burdens on your conscience or career repercussions, just throw it back and move on. We can automate that (or at least, we will be able to soon) but if we get some consensus on this, we should make our expectations of contributors explicit.

This sounds reasonable, I've seen a number of bugs where someone starts
work (or says they've started work), completely disappears, and then
other people are confused about if they can pick it up or not. We should
(obviously) be considerate with the timespan and message used here so
there are no hard feelings from contributors. A reminder at the halfway
point might help.


> [...] what constitutes the minimum viable contributor, to your corner of the Mozilla project?

>
> For core engineering, I think the answer might be "Can build Firefox on their machine, can comment on a bug, and can find us on IRC or email to talk about it."

Wow, that's a really fantastic idea. I think one of my hesitations about
marking a bug as [mentor=dolske] is that the process of bringing someone
up to a baseline level can be very time intensive and draining.
Especially when someone new comes along that needs extra hand-holding,
or simply lacks the level of competence needed for serious programming.

Which leads me to think:

* It's very useful to have a dedicated place for new contributors to go
to find help for getting-started tasks (and, by extension, people there
to help them). #introduction is already going down this path. This would
allow mentors to focus more on the code they know, and give contributors
a more consistent experience. There could be be a good feedback
mechanism here, if new contributors start helping out even newer
contributors.

* Some kind of self-guided bootcamp / tutorial would help get people to
a baseline level. For example, get them building the code, walking them
through a few simple (+fun!) modifications, then give a few challenge
ideas to tackle on their own. Stuff like "make tabs green", "add context
menu that plays a sound", "print the URL of every image that loads",
etc. If someone can do that, they're ready to dive into a Good First Bug.

Justin

[Gervase Markham]

On 08/03/13 02:16, Justin Dolske wrote:
> * Some kind of self-guided bootcamp / tutorial would help get people to
> a baseline level. For example, get them building the code, walking them
> through a few simple (+fun!) modifications, then give a few challenge
> ideas to tackle on their own. Stuff like "make tabs green", "add context
> menu that plays a sound", "print the URL of every image that loads",
> etc. If someone can do that, they're ready to dive into a Good First Bug.

Right. There could even be a badge :-) And mentors could reasonably
refuse to mentor someone through a bug who didn't have it.

Gerv

[Gervase Markham]

On 07/03/13 22:02, Michael Hoye wrote:
> sometimes it gets misflagged (flagged for review without calling out
> a reviewer, that gets mentioned a _lot_)

I'd like to ban "requests of the wind":
https://bugzilla.mozilla.org/show_bug.cgi?id=751862

Gerv

[Lawrence Mandel]

This is a really good idea to get people's feet wet before jumping into bugs. This type of simple "project" should be featured on a new contributor entry point.

Lawrence

[Till Schneidereit]

A quick note on good first bugs: I started contributing through those,
and one thing that was crucial for me was that I could take on a
couple of them without having to talk to anyone before, during, or
after fixing them.

Many people learn their way around the project much faster by getting
involved in the community and asking questions. However, for other
people that interaction is a strict addition to the efforts they have
to make to find their way through a huge new code base and to learn
how to use the tools.

It was only after I worked on a few bugs without any interaction with
the community whatsoever that I felt like I was in a position where I
could approach people and ask questions. And don't get me wrong: that
wasn't because the community felt unapproachable. I just had to build
up a solid case for myself and the nerve to do the approaching.

cf: https://blog.mozilla.org/dmandelin/2012/04/02/quietly-awesome/ and
https://staktrace.com/spout/entry.php?id=789

(and no, I'm not really an introvert, so I think the point I'm trying
to make might be all the more valid for those that are.)

(@David: Sorry for the double post, forgot to cc the list the first time around)

> On 13-03-07 5:02 PM, Michael Hoye wrote:
>>
>>
>> Hi, folks. Just a head's up, some of this wall of text is
>> engineering-specific, but I hope you'll wade through it. There's something
>> here for everyone.
>>

>> other company and it doesn't get handed off, sometimes it gets misflagged

>> (flagged for review without calling out a reviewer, that gets mentioned a

>> _lot_) and sometimes a bug just falls through the cracks of an overburdened
>> inbox.
>>
>> Sometimes, the contributor goes dark; stops responding to hails, the bug
>> goes moribund because they, for whatever reason, doesn't continue. We don't
>> have a lot of insight into why that is yet - maybe life just happens to them
>> and they need to step away, maybe they've bitten off more than they can
>> chew, maybe we could be better about clarifying what it's going to take to
>> ship a patch… lots of maybes. I'm working on them.
>>
>> I think there are two ideas that I'd like to propose here, that sort of
>> involve automation and sort of involve managing expectations and might make
>> our lives easier.
>>

>> The first is: I think we need a time limit on the activity in entry-point
>> bugs, whatever they are - two weeks seems like a sane guess, but I'm not
>> sure - after which the mentor who's supervising that work can say "This
>> isn't working, we're throwing this one back in the water", and make the bug
>> available for other contributors again. No hard feelings, burdens on your
>> conscience or career repercussions, just throw it back and move on. We can
>> automate that (or at least, we will be able to soon) but if we get some
>> consensus on this, we should make our expectations of contributors explicit.
>>

>> For the second: many of you are familiar with the idea of "minimum viable
>> product" question, "what's the smallest, simplest thing we can get out the
>> door that will make somebody's life better." The thing I'd like you (you,
>> mozilla-engineering, sure, but you "anyone who wants to work with
>> contributors on your cool stuff at Mozilla" too) to give some thought to

>> this: what constitutes the minimum viable contributor, to your corner of the

>> Mozilla project?
>>
>> For core engineering, I think the answer might be "Can build Firefox on
>> their machine, can comment on a bug, and can find us on IRC or email to talk

[Blake Winton]

I'ld like someone to add a feature to mach which would show you a
plausible reviewer for the patch you're currently writing. :)

(I might like it more if that feature was in bugzilla, so if you tried
to request from the wind, it would offer suggestions which you could
easily pick from, but putting it in mach would be a reasonable compromise.)

Later,
Blake.

[Ms2ger]

<https://bugzilla.mozilla.org/show_bug.cgi?id=774145>; patches welcome.

HTH
Ms2ger

[Gregory Szorc]

We could easily rathole into optimizing the patch submission process.
The current process is complicated to the uninitiated and full of subtle
gotchas. Here is a partial list of things that can go wrong and possible
workarounds:

* No author info in patch (Bugzilla could warn on patch upload. We could
provide a CLI tool for uploading patches that performs validation before
upload)
* Not enough lines of context in patch (ditto)
* No Mercurial headers in patch (ditto)
* Malformed summary message (ditto)
* No reviewer flagged (Bugzilla enforcement)
* Improper reviewer (possibly dead account) flagged (Bugzilla)
* Hard to find reviewer (CLI tool, maybe Bugzilla aid)
* Don't know bug component (if you start coding before filing a bug - it
happens) (CLI tool, Bugzilla "find component/bug/reviewer for patch"
functionality).
* Contributor hasn't signed contributor agreement (Mercurial push hooks,
Bugzilla flags)

Altogether, this is a massive barrier to getting code in the tree.
Compare our current code acceptance workflow to say GitHub pull
requests. Specific properties that make GitHub pull requests a superior
user experience include:

* No need to be burdened with patch formats. If you commit to your fork,
you are set. i.e. if you can use Git, you can contribute.
* No need to find a reviewer. You just submit a pull request to a shared
review queue and it is magically taken care of.

(The superior user experience outcome assumes the pure GitHub workflow -
not the GitHub + Bugzilla workflow we've imposed on ourselves.)

If we are not going to allow GitHub pull requests into m-c directly, we
should at least try to emulate some of its properties. There are
numerous options here. They include:

* Standing up a Mercurial and/or Git server that turns pushes into code
reviews.
* Ability to upload "naked" patches to a shared review queue which then
get converted into Bugzilla requests (possibly automatically, possibly
manually).
* Have Bugzilla assign reviewers for patches based on changed files
and/or bug component.
* Allow reviews to be submitted to a pool of relevant reviewers
(component watchers or something) and have Bugzilla nag when there is a
review in your component/module that has been lingering.
* CLI tools to streamline the patch lifecycle
(http://dev.chromium.org/developers/contributing-code comes to mind)
* ...

There's no shortage of possibilities for us to try. We just need to
commit to some of them.

[David Rajchenbach-Teller]

On 3/8/13 3:16 AM, Justin Dolske wrote:
> I guess the implication here is that we would have a set of bugs aimed
> at students? Sounds like a fine idea, I've just not heard of it before.

Well, we have these:
https://github.com/Yoric/Mozilla-Student-Projects/issues

>> The first is: I think we need a time limit on the activity in
>> entry-point bugs, whatever they are - two weeks seems like a sane
>> guess, but I'm not sure - after which the mentor who's supervising
>> that work can say "This isn't working, we're throwing this one back in
>> the water", and make the bug available for other contributors again.
>> No hard feelings, burdens on your conscience or career repercussions,
>> just throw it back and move on. We can automate that (or at least, we
>> will be able to soon) but if we get some consensus on this, we should
>> make our expectations of contributors explicit.
>
> This sounds reasonable, I've seen a number of bugs where someone starts
> work (or says they've started work), completely disappears, and then
> other people are confused about if they can pick it up or not. We should
> (obviously) be considerate with the timespan and message used here so
> there are no hard feelings from contributors. A reminder at the halfway
> point might help.

For the purpose of helping me track down the progress of student bugs, I
have patched the bugzilla dashboard:

http://yoric.github.com/bugzilla-dashboard/#username=dte...@mozilla.com&mentorname=Yoric


Cheers,
David

--
David Rajchenbach-Teller, PhD
Performance Team, Mozilla

[Jeff Hammel]

On 03/08/2013 07:39 AM, Till Schneidereit wrote:
> <snip/>

> It was only after I worked on a few bugs without any interaction with
> the community whatsoever that I felt like I was in a position where I
> could approach people and ask questions. And don't get me wrong: that
> wasn't because the community felt unapproachable. I just had to build
> up a solid case for myself and the nerve to do the approaching.
>
>

I'd note that contributors tended to be of two camps here:
1) overly cautious, building up knowledge and THEN launching into questions
2) asking about everything up front; e.g. "How do you do this in python?"

While there's nothing wrong with the second approach, it quickly
degenerates to teaching them programming, which while a worthwhile goal,
isn't the purpose of mentored bugs.

Really, I'd love to see something between the two; that said, cautious
is much more easy to deal with.

[Liz Henry]

On 3/7/13 2:02 PM, Michael Hoye wrote:


>
> The first is: I think we need a time limit on the activity in
> entry-point bugs, whatever they are - two weeks seems like a sane
> guess, but I'm not sure - after which the mentor who's supervising
> that work can say "This isn't working, we're throwing this one back
> in the water", and make the bug available for other contributors
> again.

I would like to set up those contributors -- the ones who sign up to fix
a good first bug or student bug, but don't follow through -- to be
funnelled into bugmastering and triage. If there is an automated
response as people are un-assigned bugs, they could be redirected to the
resource for bugmasters. This could be offered as a way to become more
familiar with Mozilla's infrastructure, reading and discussing bugs, and
looking at code and builds without having to fix stuff.

If I could get their emails as they fall into the "dropped a good first
bug or student project" bucket, then I could possibly include them in
invitations to bug days and triage.

In short I would like to offer the bugmastering community as a
mentoring path that can bring people into positive contributions with
(perhaps) less learning curve or time commitment, and give them more
confidence and experience with Mozilla.


- lizzard






--
Liz Henry
Bugmaster
lhe...@mozilla.com