Engineering Effectiveness Newsletter (May 2023 Edition)
Dave Lawrence
Welcome to the May edition of the Engineering Effectiveness Newsletter! The Engineering Effectiveness org makes it easy to develop, test and release Mozilla software at scale. See below for some highlights, then read on for more detailed info!
Highlights
Translations has been enabled in Nightly and early Beta, please test it and file bugs!
Contributors
Tim van der Meij (timvandermeij)
Jonas Jenwald (Snuffleupagus)
Bungeman
Anurag Bhandari (WhiteWolf47)
Emeka Onwuepe (Emeka-Onwuepe)
Kenneth Chew (kthchew)
Detailed Project Updates
Bugzilla and Bugbug
Suhaib added support for notifying about potential uplift worthy patches for mobile products.
The old severity (major, minor, critical, …) to new severity (S1, S2, …) migration has been completed.
Marco relaxed the rules to close no longer happening intermittent bugs, which will allow closing more than 2000 bugs that were previously lingering around with no activity.
Suhaib implemented a new BugBot rule to send a weekly list of security bugs that need to be unhidden (as they are duplicates of already unhidden bugs). The next step is automating the unhiding.
Anurag Bhandari implemented calibration for the bugbug regressor model.
Emeka Onwuepe made two improvements (1, 2) to the bugbug commit classifier.
David Lawrence added support for linking comment tags to web pages (like wiki pages, help, etc)
Build System and Mach Environment
Build of firefox for android has been documented https://firefox-source-docs.mozilla.org/mobile/android/fenix.html
Builds now require at least clang 7 (up from clang 5.0) and at least clang-cl 9 (up from clang-cl 8)
Builds now require at least gcc 8 (up from gcc 7)
Builds now require at least MSVC 2019 (up from MSVC 2017)
Builds now require at least macOS SDK 13.3
Builds now require at least rustc 1.66.0
Builds don't require Perl anymore (except when targeting 32-bits ARM).
External contributor :smurfd added support for ./mach bootstrap for source tarballs (ie: without having git or hg).
Serge Guelton completed work removing the requirement for unified builds, and blogged about it https://serge-sans-paille.github.io/pythran-stories/how-unity-builds-crept-into-the-firefox-build-system.html
Serge Guelton completed a deep investigation into how compiler flags impact run-time performance, with the final recommendation of using FullLTO combined with LTO to yield faster binaries on Windows and Linux.
CI and Treeherder
Sccache: Version 0.5.0 released
We removed a lot of useless files on our Taskcluster storage, saving a lot of money every month.
We also filed follow-up bugs to set the expiration for these files to be shorter, so we keep these savings long term.
ahal implemented a new set of transforms in Taskgraph called from_deps. These aim to make it easy to create follow-up tasks and will replace the multi_dep loader.
aerickson and ahal are working on getting tests running in a Wayland environment
Crash Management
All environment management functions (getenv(), setenv(), etc…) now have interposers that provide thread-safe access on both Linux and Android. This prevents a wide range of crashes our users were experiencing. This work on the interposers also had the side-effect of making stack overflows catchable in Android main-process crashes.
Crash instruction disassembly now works on 32-bit x86 builds where it had been disabled due to a regression in the minidump generation code
Several crash analysis features were added to the stackwalker: crashes caused by NULL pointer accesses are now clearly reported , native debug information can be used for stack-walking and heuristics were added to detect crashes caused by bit-flips (often caused by flaky hardware) and accesses to guard pages (which might indicate potential security issues)
Lint, Static Analysis and Code Coverage
The enabling of Prettier on JSON files has now landed.
Gijs has landed stylelint for css linting.
This will flag up errors in your CSS, like duplicate properties, invalid syntax, missing generic font families, etc. In some cases it can auto-fix things.
We will expand this in the next few weeks/months to cover conventions in use in Firefox (e.g. using “em” for font-size over “px”, preferring logical over physical margin/padding/border/float definitions, using “0” without a unit for sizes, etc.)
This is not a formatter like prettier, so it doesn’t currently adjust whitespace, indenting etc.
./mach lint . now works in parallel mode for faster linting. Previously only ./mach lint * would work, but that didn't pick up the top-level dot files.
Next up, is to upgrade Prettier and enable it for production HTML files.
The upgrade does change a significant number of JS files due to changes in Prettier's formatting.
We decided to put off enabling it on test files for now, due to the large amount of files affected.
OS Integration and Security
We now have a methodology for measuring the impact of antivirus products on CPU usage when Firefox is running and a meta bug to track the work on this topic. QA has successfully applied the methodology on two Nightly builds to help us measure the effect of a specific patch on AV-induced CPU usage. The next step is to collect similar data recurrently during release cycles.
cargo-vet has added a number of user conveniences to streamline auditing, and is preparing for a release following an early announcement from Google regarding their use of the tool to audit their dependencies.
PDF.js
Calixte implemented a fallback on system fonts when a font isn’t embedded in the PDF. It helps to improve the rendering on Linux and Android for missing fonts. Jonas fixed many bugs unlocked by this change.
Calixte added support to run Nimbus experiments with pdf.js on Firefox for Android, and Telemetry collection for buttons.
Calixte simplified the way to smooth curves when drawing in order to have something more natural and a bit less aggressive.
Tim van der Meij introduced a GitHub Actions workflow to publish the pdf.js website (first step to moving off of our custom CI running on AWS).
bungeman introduced a better approximation for gradient color stops.
Firefox Translations
We've been working to migrate the Firefox Translations addon directly into mozilla-central, see the meta bug for all the progress so far!
… and Translations has been enabled in Nightly and early Beta, please test it and file bugs!
Greg Tatum created an Are We Translations Yet page to track the engineering work.
Greg Tatum landed about:translations.
Erik Nordin has landed new ML based infrastructure for detecting languages. The screenshot below shows the language code for the text that is detected along with a probability.
Serge Guelton implemented a replacement of intgemm (8-bit integer matrix multiplication using xsimd when possible) that is easier to integrate into Firefox and easier to maintain
Greg Tatum ported the full page translation feature from the addon, and got it working with the JSActor system so that it works on both mobile and desktop. This means that webpages can be fully translated, and kept translated.
Greg Tatum implemented the translations popup to power full page translations.
Erik Nordin added source documentation for the Firefox Translations project, including documenting the process of building the fastText (language identification) models for Firefox, defining the process of versioning machine-learning model and WASM upgrades across breaking and non-breaking changes.
Erik Nordin added the ability to distinguish between which translation models are in beta or not based on the versioning documentation in the above bullet point.
Ben Hearsum has made significant progress moving the language training pipeline into Taskcluster and GCP. When completed, this work will allow us to better optimize and scale our training jobs.
Phabricator , moz-phab, and Lando
Zeid fixed a bug in Lando that was causing deadlocks during deployments that had certain types of database migrations (bug 1832083).
Zeid added filesystem caching support in Lando, making local testing and development faster (bug 1832142).
Zeid simplified and improved landing worker functionality by implementing a proper many-to-many relationship between landing jobs and revisions, reducing code complexity and improving versatility (bug 1825277).
May Lando deployments (see release notes): release-39, release-40, release-41, release-42.
Thanks for your patience during Lando maintenance downtimes!
Zeid added the latest diffID to Phabricator’s differential.revision.search endpoint, which is now returned in the response (bug 1798737) removing the need for additional queries to fetch the diffID from a “phid”.
Connor Sheehan linkified differential revision text in Landing warnings, making it easier to go to revisions with landing warnings to address them.
Release Engineering and Management
Heitor Neiva got Firefox Nightly notarized with rcodesign (bug 1774273). Shipping Firefox on macOS requires our builds to be submitted to Apple’s Notary service. Otherwise macOS users get a scary warning about the origin of the binary. Moving to rcodesign (instead of Apple’s tool) reduces RelEng’s dependency on hard-to-maintain Mac machines.
Geoff Brown optimized the infamous external-gradle-dependencies tasks on Android (bug 1820122) by finding a great compromise. These tasks are notoriously long to execute and Geoff halved their duration. He found a sweet spot between the duration of the tasks and the number of gradle dependencies that get cached.
Ben Hearsum, Heitor Neiva, and Hal Wine rotated Mozilla’s GPG signing key. This key is used to sign a lot of Mozilla’s artifacts including Linux binaries, source files, checksums, etc. We rotate it every 2 years and we strictly follow a security procedure to ensure no key gets compromised.
Andrew Halberstadt worked with the Application Services team to help set up a release promotion pipeline + beetmover tasks in their CI.
Julien Cristau implemented a better way to reference GitHub commits on mozilla-central (firefox-android#2197). This will be used as part of the migration of the Android repository to mozilla-central.
Gabriel Bustamante made Mozilla’s .deb package better integrated with Linux Desktop environment. Now, pretty names are displayed, icons are shows, mime-types are correctly associated (bug 1824327)
The Mozilla VPN is now officially part of RelEng’s releaseduty process #releaseduty - Escalating release issues, thanks to coordination between Adrienne Davenport and Johan Lorenzo
Version Control
Connor Sheehan made the robots.txt file for hg.mozilla.org much more restrictive in an effort to reduce the load impact of crawlers.
Connor Sheehan and Chris Knowles added User-Agent string based banning to hg.mozilla.org to ban service abusers.
mozregression
Zeid improved speed of cross-platform requirements generation by running jobs in parallel (bug 1810224)
Kenneth improved the UI usability in dark mode (bug 1719424)
Thanks for reading and see you next month!
--