Evaluating the performance of new features (plain text version)

[Vladan Djeric]

(I forgot step 0, don't post HTML messages to a newsgroup ;))

Hi all,

There are a lot of good tools available now for studying Firefox
performance, and I think a lot of them are not well known, so I put
together a list of steps to follow when evaluating the performance of
your next Firefox feature.

1. Make sure to test your feature on a low-end or mid-range Windows computer

- Our dev machines are uncommonly powerful. Think machines with
spinning hard drives, not SSDs. Also make sure to test on Windows, as
it is used by the vast majority of our users.
- The perf team, fx-team, and gfx team have Windows Asus T100 tablets
[1] available in multiple offices just for this purpose. Contact me,
Gavin, or Milan Sreckovic if you need one.

2. Ensure your feature does not touch storage on the main thread,
either directly or indirectly

- If there's any chance it might cause main-thread IO, test it with
the Gecko profiler [2]. The profiler now has an option to show you all
the IO done on the main thread, no matter how brief it is.
- Also be careful about using SQLite [3]

3. Make sure to add Telemetry probes that measure how well your
feature performs on real user machines

- Check the Telemetry numbers again after your feature reaches the
release channel. The release channel has a diversity of configurations
that simply don't exist on any of the pre-release channels.
- You can check for regressions in the Telemetry dash [4] or you can
ask the perf-team to show you how to do a custom analysis (e.g.
performance on a particular gfx card type) using MapReduce [5] or
Spark [6]
- The learning curve can be a bit steep, so the perf team can do
one-off analyses for you.
- There are additional performance dashboards, listed in the "More
Dashboards" sidebar [4]
- Always set the "alert_mails" field [7] for your histogram in
Histograms.json so you get automatic e-mail notifications [8] of
performance regressions and improvements.
- Ideally, this email address should point to an alias for your team.
- Note that the Telemetry regression detector [9] has an extremely low
false-positive rate so you won't be getting any emails unless
performance has changed significantly.

4. Keep an eye out on the Talos scores

- The Talos tests are much less noisy now than they used to be, and
more sensitive as well. This is thanks to Avi Halachmi's, Joel
Maher's, and others' efforts.

How Talos sheriffing works:
- Partly as a result of the test noise improvements, we now have a
stricter Talos sheriffing policy. The patch author has 3 business days
to respond to a Talos regression bug (before getting backed out), and
two weeks to decide what to do with the regression.
- Joel Maher will file a regression bug against you if you regress a Talos test.
- The list of unresolved regressions in each release is tracked in the
meta bugs [10]
- Joel tracks all the improvements together with all the regressions
in his dashboard [11]

Diagnosing regressions:
- If you cause a regression that you can't reproduce on your own
machine, you can capture a profile directly inside the Talos try
environment [12]
- MattN has an excellent tool for comparing the scores of two Talos pushes [13]
- Some Talos tests can be run locally as extensions, others may
require you to set up a Talos harness [14]. Instructions for doing
this will be provided in the Talos regression bugs from now on.
- The graph server [15] can show you a history of test scores and test
noise to help you determine if the reported regression is real.
William Lachance is working on a new & improved graphing UI for
treeherder.

5. Consider writing a new Talos test
- Add a new Talos test [16] if the performance of your feature is
important and it is not covered by existing tests. The Perf team would
be happy to help you design a meaningful and reliable test.
- Make sure your test measures the right things, isn't noisy and that
it is is able to detect real regressions

Thanks!


Links:
1. http://www.amazon.com/Transformer-T100TA-C1-GR-Detachable-Touchscreen-Laptop/dp/B00K0G2XA4
2. https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Reporting_a_Performance_Problem
3. https://wiki.mozilla.org/Performance/Avoid_SQLite_In_Your_Next_Firefox_Feature
4. https://telemetry.mozilla.org/
5. http://mreid-moz.github.io/blog/2013/11/06/current-state-of-telemetry-analysis/
6. http://robertovitillo.com/2015/01/16/next-gen-data-analysis-framework-for-telemetry/
7. https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Adding_a_new_Telemetry_probe#Declaring_a_Histogram
8. https://groups.google.com/forum/#!forum/mozilla.dev.telemetry-alerts
9. http://mozilla.github.io/cerberus/dashboard/
10. For example https://bugzilla.mozilla.org/show_bug.cgi?id=1122690
11. http://alertmanager.allizom.org:8080/alerts.html?showAll=1
12. https://wiki.mozilla.org/Buildbot/Talos/Profiling
13. http://compare-talos.mattn.ca/
14. https://wiki.mozilla.org/Buildbot/Talos/Running#Running_locally_-_Source_Code
15. http://graphs.mozilla.org/graph.html
16. https://wiki.mozilla.org/Buildbot/Talos/Misc#Adding_a_new_test