Sheriffing Newsletter, End of September Edition

[Carsten Book]

Hi,

To give a little insight into our work and make our work more visible to
our Community we decided to create a monthly report of what's going on in
the Sheriffs Team.

If you have questions or feedback just let us know!

In case you don't know who the sheriffs are or if there are current issues
on the tree, see:
https://sheriffs.etherpad.mozilla.org/sheriffing-notes

Topics of this month!

1. How-To article of the month
2. Get involved
3. Statistics
4. Orange Factor
5. Meeting notes
6. Contact

1. How-To article of the month and notable things!

First: Welcome Nigel Babu (irc: nigelb) our latest addition to the Sheriffs
Team!

Also a few words about checkin-needed requests since we got some questions
during the month.

-> Work is being done on an autolander system that lands your patches to
mozilla-inbound for example. However that system is not ready yet, so all
checkin-needed requests are being handled by a human and not automated.

-> When you have multiple check-ins that need to land, it would be helpful
if you could indicate in the checkin-needed request which dependencies the
bug has, or which parts of the patches you want to get landed (like part 3
to 4 etc)

-> It would be super helpful if your commit message on the patch you want
to get checked-in is complete with bug number, title and reviewer name.
This helps a lot!

2. Get involved!

Are you interested in becoming a Community Sheriff and helping oversee
test results? Let us know!

3. Statistics

Intermittent Bugs filed this month [1]: 400
Intermittent Bugs closed this month [2]: 214

For Tree Closing times and reasons see:
http://futurama.theautomatedtester.co.uk/

4. Orange Factor

Current Orangefactor [3]: 8.77

5. Meeting Notes:

We Sheriffs hold regular meetings. Our notes can be found here:

https://sheriffs.etherpad.mozilla.org/meeting-2015-09-14
https://sheriffs.etherpad.mozilla.org/meeting-2015-09-21

6. How to contact us

There are a lot of ways to contact us. The fastest one is to contact
the sheriff on duty (the one with the |sheriffduty tag on their nick
:) or by emailing sheriffs @ mozilla dot org.

Cheers,

[1]: http://mzl.la/1LgxbKc
[2]: http://mzl.la/1PJ287C
[3]: https://brasstacks.mozilla.com/orangefactor/

[Ehsan Akhgari]

On 2015-09-28 7:26 AM, Carsten Book wrote:
> 3. Statistics
>
> Intermittent Bugs filed this month [1]: 400
> Intermittent Bugs closed this month [2]: 214

In the past couple of weekends, I spent some time fixing the worst
offenders in intermittent oranges according to OrangeFactor, and the
number 214 above was *shocking* to me since almost none of the bugs I
looked at had any activity on them. Skimming over
http://mzl.la/1PJ287C, it seems that many of these are bugs that were
closed because the failures weren't happening before, which makes it
confusing to mix that pool with the actual number of bugs that were
fixed as a result of a patch landing.

Can we somehow improve the statistics here to measure the actual bug
fixes? A change as simple as filtering out "nob...@mozilla.org" as the
assignee of the bug narrows the list down to 45 bugs, but that is
probably inaccurate since the merge tool had not adjusted the assignee
of bug 703773 for example. But just filtering based on the assignee
seems like a better way to get data closer to reality here.

Cheers,
Ehsan

[Wes Kocher]

What if the query was changed to only count bugs resolved 'fixed'? I
believe all of the inactive intermittent bugs get closed as worksforme or
invalid?

On Mon, Sep 28, 2015 at 10:45 AM, Ehsan Akhgari <ehsan....@gmail.com>
wrote:

[Ryan VanderMeulen]

Included auto-closed bugs in that count sounds misleading at best.

[joel maher]

For the opened bugs it would be nice to know to focus on ones that have >1
instance, that means that there is a much higher chance to reproduce the
bug rather than letting hundreds (or even 1000+) of test runs go by without
seeing another instance of the failure.

-Joel

[Ehsan Akhgari]

On 2015-09-28 1:54 PM, joel maher wrote:
> For the opened bugs it would be nice to know to focus on ones that have
> >1 instance, that means that there is a much higher chance to reproduce
> the bug rather than letting hundreds (or even 1000+) of test runs go by
> without seeing another instance of the failure.

My general rule is that to have the most impact, people who want to help
should go to http://brasstacks.mozilla.com/orangefactor/ and help close
the bugs from the top of the list down to the bottom. :-) That is the
recipe to get a decreasing slope on the orange factor graph.