Engineering Effectiveness Newsleter (July 2022 Edition)
Andrew Halberstadt
Welcome to the July edition of the Engineering Effectiveness Newsletter! The Engineering Effectiveness org makes it easy to develop, test and release Mozilla software at scale. See below for some highlights, then read on for more detailed info!
Highlights
Autonag received many new features, including auto-rotating triage owners, highlighting inactive reviewers or fuzzing problems and less bugmail spam!
A promising fix has landed which appears to reduce Windows OOM crashes by as much as 20%!
Most of Firefox’s test tasks have been migrated from AWS to GCP, resulting in a large cost savings!
Many improvements landed to the Code Review bot, including hints if an issue may not be an author’s fault, colorized output, a fix for warnings and less noise!
Detailed Project Updates
Bugzilla and Bugbug
Suhaib implemented several new features for autonag!
Ability to automatically update the triage owner on Bugzilla based on a rotation calendar.
Highlight patches that are waiting for review from inactive or unavailable reviewers.
Ability to perform actions without triggering bugmail! This will reduce noise when the bot is performing actions that do not require attention.
Bring attention to bugs that cause issues for fuzzing.
dkl implemented changes to the See Also field in Bugzilla which will now take any properly formatted URL in addition to local bug IDs.
To reduce spam in Bugzilla, users without "editbugs" can no longer make changes to bugs in the Graveyard classification (ie. bugs associated with archived products).
CI and Treeherder
ahal, jlorenzo and others released Taskgraph 2.0! See the migration page for tips on how to upgrade from 1.x
ahal revamped Taskgraph’s documentation by writing new content, porting content from Firefox Source Docs, and giving it a nicer theme.
masterwayz migrated most of Gecko’s test tasks from AWS to GCP.
Valentin performed a major Django 4.0 upgrade in Treeherder.
Crash Management
Alexandre Lissy made Linux system symbol scrapers run automatically on TaskCluster. Symbols and unwinding information for Firefox builds packaged by major Linux distributions as well as Firefox dependencies will now be automatically scraped every 24h.
Alexandre Lissy added a task that reprocesses crashes that were missing system symbols after we scraped new ones, thus ensuring that all crash reports are properly symbolicated, even after the fact. This works on all platforms for which we have working scrapers (Windows and Linux currently).
Lint, Static Analysis and Code Coverage
Many great improvements landed in the Code Review bot!
vrigal added a hint if the issue comes from a parent revision.
EvaBardou ensured the bot will no longer post previously found issues on newer Phabricator revisions.
EvaBardou fixed a bug in the backend preventing warnings from being displayed.
EvaBardou colorized the bot’s comments according to their meaning.
OS Integration and Security
The Windows Spotlight team continues to take a targeted approach to addressing high-impact issues for our Windows users. Main areas of focus in July were:
OOM crashes have been identified as a source of anywhere between 30-50% of crashes on Windows. rkraesig, in coordination with the broader Windows Spotlight team, has taken a deep-dive into possible remediations, with one particularly promising approach having the potential of reducing these OOM crashes by as much as 20%. We are now awaiting the numbers to evaluate the actual effectiveness of this fix.
bobowen has landed a change to use the WER Runtime Exception module which allows us to catch crashes that occur earlier in startup than ever before!
Looking forward, the Windows Spotlight team will also start addressing high-impact issues affecting touch devices.
PDF.js
calixte made accessibility improvements and added touch support to PDF editing!
Release Engineering and Management
Starting with Firefox 103, the Release Management team are releasing a planned dot release 2 weeks after the previous release go-live.
Fx-Trains now offers a calendar view of upcoming releases
gbrown reviewed and fixed openh264 builds and signing, and added notarization of openh264 on Mac.
Guided by :RyanVM, :jcristau and :jlorenzo improved the automation around our Android products. Starting with Gecko 104:
Android-Components gets released off of https://shipit.mozilla-releng.net/ instead of Github Releases
Fenix/Focus are not immediately shipped anymore. We now have a 2-step process where builds are promoted and sent to our testing population before shipping them to Github or archive.mozilla.org.
Version numbers now follow the gecko scheme. This means 104.0.0-beta.1 is now called 104.0b1.
L10n bumps and Android-Components bumps are now automatically approved and merged if they passed CI, on the release branches. This used to work on the main branches only.
Other
ahal added a new :searchfox: role to firefox-source-docs which can be used to more conveniently link to Searchfox.
Thanks for reading and see you next month!