Welcome to the June (and first) edition of the Engineering Effectiveness Newsletter! The Engineering Effectiveness org makes it easy to develop, test and release Mozilla software at scale. See below for some highlights, then read on for more detailed info!
Content process crashes generated via Windows Error Reporting are now being intercepted!
Early data suggests around 10% of crashes were previously being missed
Many new bugs being found by these crashes, including security sensitive ones
We are possibly the first FOSS project to do this
The minimum version of Python is now 3.6, no more Python 2!
All mach commands were converted to Python 3
Feel free to start using 3.6+ features like f-strings and async/await in your Python
A new `about:third-party` page has been created for shedding light on code injected by third parties
Bugs in these modules often get blamed on Mozilla
Previously these were invisible to users, now they can be used to help diagnose support issues
Google’s Fuzzilli (a coverage-guided fuzzer) has been integrated into our JS engine and deployed on our fuzzing infrastructure!
Will help find additional bugs and vulnerabilities in the JS engine
There are some exciting changes coming with respect to lint and formatting in Phabricator!
Automatic code formatting on landing, currently only Python is supported but support for more languages coming in H2
Warnings are now displayed by reviewbot. These are optional fixes which won’t cause a back out. This is the first time they are being surfaced.
Speaking of Phabricator, the e-mails it generates have gotten a major overhaul!
They’re now less noisy, less cluttered and packed with only the most relevant context, so you can tell at a glance exactly what is needed from you
Now enabled as the default setting
marco and jcristau made some improvements to autonag / bugbug
Better support for detecting regression bugs
Catching some security fixes that were escaping the uplift and advisories process
ksenia added the ability to build, deploy and serve bugbug machine learning models on GitHub data
dkl added the ability for privileged users to update triage owners in Bugzilla
This means teams can self-serve triage ownership changes, which is especially useful if they have a triage owner rotation
dkl also added client and server side enforcement of triage requirements
For example, severity will now automatically be set to S2 for crash bugs
See here for more of the enforced rules
mhentges, ahal and others finished migrating all mach commands to Python 3
The minimum required Python version was also upgraded to 3.6!
Thanks to Surajeet who made many contributions removing Python 2 compatibility shims
mhentges also added the ability to source dependencies from PyPI via virtualenv_requirements files
This is an early step towards a larger goal of re-working how Python dependencies are managed in mach commands
glandium turned on --enable-bootstrap by default
This means toolchains and other build dependencies will automatically be updated without the need to run `mach bootstrap`
andi enabled telemetry to determine the number of developers that use VSCode over ssh
mhentges removed ~150,000 lines of unnecessarily vendored Python tests and metadata!
marco improved the test selection algorithm and enabled it for mach try
Running `mach try` should now provide much better results by default
sclements worked with oremj to migrate Treeherder from heroku to gcp/cloudOps with very little down time
sclements also hit a milestone with Push Health
Push Health is a developer-friendly alternative to Treeherder’s default jobs view
Bug reports and feature requests welcome!
andi added Firefox builds using Clang trunk for linux64
These are currently running as a daily cron task on central
This will allow us to identify issues caused by upstream clang changes much sooner
andi also added awsy and browsertime performance tests to monitor regressions in Firefox built with clang-trunk vs the official clang toolchain
Thanks to masterwayz’s contribution to finish migrating tests from OSX 10.14 -> 10.15
jmaher moved the majority of Windows 7 32 bit tests to Windows 10 32 bit
jmaher also supported the WebRender team in their efforts to expand test coverage
As well as starting to turn off non WebRender tests
ahal ported the standalone taskgraph module to Python 3
Fenix’s taskgraph has also been migrated, but other consumers still left todo
ahal also enabled many more tasks in support of Fission
gsvelto enabled interception of content process crashes via Windows Error Reporting
Around 10% of crashes were being missed, no longer!
OOM crashes intercepted via WER are now properly flagged
All crashes intercepted via WER are now labelled as such
gsvelto also added names for Windows error values to the oxidized stack walker
msirringhaus added recording native thread names in Linux minidumps
gankra has added support to the crash reporter for using macos' compact unwinding info
Ideally this will result in better stack traces on macos
Results currently mixed, have found at least one bug in the implementation
decoder integrated fuzzilli (a coverage-guided JIT fuzzer by Google P0) into our JS engine and deployed it in our internal fuzzing infrastructure.
tsmith added support to handle OOMs and hangs to Grizzly
Existing fuzzers using Grizzly will get this automatically
Thanks to dnaka91’s contribution which adds support for grcov to output coverage badges
Great for projects who want static coverage reports without relying on external services.
Thanks to geovie’s contribution implementing the Cobertura output format for grcov
Also thanks to svenstaro’s contribution making it more readable
As well as svenfoo’s contribution with a fix for larger projects
marco updated code review bot to show all warnings found by mozlint-based linters
Since warnings are optional, this is the first time they are being surfaced to developers by default!
tkikuchi added a new about:third-party page to visualize third-party modules injected into Firefox processes
flopez also helped update SUMO docs to explain how to use it
This sheds some light on previously invisible modules injected by third parties
bdahl added support for the standard 14 fonts (which are not always embedded in PDF files)
bdahl also added support for tagged PDFs to improve screen reader accessibility
calixte added support for XFA forms
These are still used by several governments (like Canada)
Test it out by setting “pdfjs.enableXfa” to true in about:config.
mhentges deployed the new Phabricator Emails implementation to all developers
This provides much clearer and more actionable e-mails
zeid worked with ops to enable continuous deployment of Lando
Will be used as a model for other applications
sheehan implemented automatic formatting of code at landing time
Initially limited to Python, but more languages on the way in H2
bhearsum and aki got Mozilla Online Fenix builds working in automation
These will be shipping in China
aki automated l10n cross-channel
This aggregates all shipping en-US Firefox and Thunderbird strings for localizers to translate
aki mitigated multiple nightlies off the same revision
pascal successfully managed the release of MR1
sclements made some UX improvements to shipit that surfaces metadata about releases
More work here to come
Engineering Effectiveness is hiring! If you know of anyone who would be a great fit for any of the following roles, consider referring them for a bonus.
Thanks for reading and see you next month!