Engineering Effectiveness Newsletter (Q3 2025 Edition)

8 views
Skip to first unread message

Marco Castelluccio

unread,
Oct 7, 2025, 2:33:23 PM (3 days ago) Oct 7
to dev-pl...@mozilla.org, Firefox Dev

Welcome to the Q3 edition of the Engineering Effectiveness Newsletter! The Engineering Effectiveness org makes it easy to develop, test and release Mozilla software at scale. See below for some highlights, then read on for more detailed info!

Highlights

  • David Lawrence implemented changes to Phabricator and Bugzilla to automatically set the reviewer of a new revision based on a rotation group. This is to ensure that revision reviewers against the same areas of code are more evenly distributed. More information at https://wiki.mozilla.org/Phabricator/ReviewerRotation.

  • Heitor Neiva switched build tasks to AMD (C3D) compute types, introducing a 28% speed improvement on CPU-bound tasks compared to Intel (C2). This resulted in a 50% reduction in compute engine costs. Test tasks were also migrated showing good results.

    • Scroll down for more build and test perf improvements!

  • It’s now possible to add/edit/delete comments in PDFs:

Screenshot of a PDF opened in Firefox, with text highlighted and a popup open to add a comment Screenshot of a PDF opened in Firefox, with text highlighted and a popup open with a commentScreenshot of a PDF opened in Firefox, with a sidebar showing a list of comments

  • We released a bunch of new models for Translations, both covering additional languages and improving already existing ones

    • In parallel, we made memory and performance improvements to allow running larger and better models on both Desktop and Mobile.

  • Connor Sheehan migrated the uplift request assessment form from Phabricator to Lando, streamlining the process down to a single step and paving the way for further improvements to the workflow (bug 1982619).

  • The Release Management team have been facilitating the transition to a weekly release cycle for iOS by coordinating timelines, updating processes, and supporting engineering/QA teams throughout the migration.

Detailed Project Updates

Bugzilla and Bugbug

Build System and Mach Environment

  • The minimum supported versions of clang and GCC were bumped to 17 and 10, respectively. The minimum supported version of libstdc++ was bumped to 10, allowing us to start to experiment with C++20 in limited parts of the codebase.

  • Linux shippable builds are now 13% faster to complete thanks to fat-lto. Linux instrumented builds are now 10% faster to complete thanks to skipping redundant plugin checks.

  • Artifact build jobs finish 20% faster, thanks to no longer repackaging crash reporter symbols.

  • Linux non-shippable builds get packaged faster, thanks to no longer using extreme compression flags.

Crash Management

  • Alex Franchuk worked with Ben Wu to create the telemetry.firefox_crashes view in BigQuery, which combines all crash pings across the various platform- and project-specific views, deduplicates pings, and has other conveniences to make using the data less error-prone.

  • The Legacy Telemetry crash ping has been removed!

Firefox-CI, Taskcluster and Treeherder

  • Heitor Neiva switched build tasks to AMD (C3D) compute types, introducing a 28% speed improvement on CPU-bound tasks compared to Intel (C2). This resulted in a 50% reduction in compute engine costs. Test tasks were also migrated showing good results.

  • Julien Cristau and Joel Maher migrated X11 test tasks from Ubuntu 18.04 -> 24.04.

  • Joel Maher (and engineers from Mobile) migrated android emulator tests from Android 7.0 -> 14.0.

  • Ryan Curran (with help from Joel Maher) moved all macstadium machines to Mozilla Datacenter.

  • Joel Maher and Andrew Erickson migrated most Android Hardware jobs from SmartBear -> LambdaTest.

  • Florian Quèze made profiler data collected in CI more discoverable from Treeherder

  • Florian Quèze made treeherder failure summaries, push results and log viewer load much faster by doing network requests in parallel and limiting component updates.

  • Florian Quèze made the tables in the treeherder intermittent failure main view and bug details view show a lot more useful information at once.

  • Florian Quèze added a shift+l new keyboard shortcut on treeherder to quickly open the raw log of a task. Did you know treeherder has many useful keyboard shortcuts documented on the user guide?

  • Florian Quèze published new dashboards showing the time xpcshell and build jobs take, and the failure rate of various test harnesses over time.

  • Florian Quèze optimized the use of CPU cores during test jobs:

  • Thanks to Whimboo and Haik for troubleshooting and providing a fix, jobs on macOS 14 workers in CI are now running much faster due to cleaning up some OS level caching.

  • Ben Hearsum made Decision task & ./mach taskgraph perf improvements.

  • Ben Hearsum refactored mochitest definitions into their own kind. This somewhat simplifies test configuration, and paves the way for future taskgraph performance improvements.

  • Joel Maher added log parser intermittent detection as part of Treeherder log parsing.  Upcoming changes to Push Health will utilize this to hide the majority of the intermittents.

  • Andrew Halberstadt fixed taskgraph build-image to use the same mechanism used by tasks in CI. This means any in-tree Docker image can now be built locally out of the box.

  • Andrew Halberstadt implemented many improvements to taskgraph load-task. It’s now possible to run any task that uses Docker in a local container.

    • You can easily test locally generated task definitions, as well as local changes to Docker images.

    • See the Taskgraph documentation for how to get started.

  • Abhishek Madan made various improvements to Taskgraph

    • Replaced Taskcluster REST API calls with the taskcluster package, simplifying integration and reducing reliance on manual API calls.

    • Improved Python version checks in CI by switching to the uv command.

    • Cleaned up Taskgraph configuration by removing exceptions for VCS root property and eliminating hardcoding from Docker image directory and kinds.

  • Matt Boris fixed a bug in Azure spot instances used by Fuzzing Taskcluster workers. Before then, when Azure terminated the instance, we had to wait 20 minutes before Taskcluster understood the Azure worker disappeared. Now it happens within a minute or so. The fix improved the situation in 90% of the previously observed errors. 

Lint, Static Analysis and Code Coverage

  • Code Review Bot detects a better suitable base revision for patch stacks which partially landed.

PDF.js

  • It’s now possible to add/edit/delete comments in PDFs. The feature will ship in Firefox 145!

Screenshot of a PDF opened in Firefox, with text highlighted and a popup open to add a commentScreenshot of a PDF opened in Firefox, with text highlighted and a popup open with a commentScreenshot of a PDF opened in Firefox, with a sidebar showing a list of comments


Firefox Translations

Screenshot of the old about:translations page in Firefox

  • After

Screenshot of the new about:translations page in Firefox, greatly improved compared to the old design

  • (Bug 1982594) Erik Nordin concluded a 3-month Outreachy internship with Isaac Briandt, which has resulted in our Translations models being served and stored on devices in zstd-compressed format. This is a reduction of 33.83% of the cumulative size of our hosted artifacts (2.45 GB to 1.62 GB). Our 32MB models now take up only 23MB on disk, which has a number of benefits:

    • Reduced time and bandwidth for users to download our models.

    • Reduced space taken up on the user's device after downloading.

  • (Bug 1820252) Erik Nordin updated the Full-Page Translation algorithms to now prioritize content based on proximity to the device's viewport. This greatly improves the felt-performance experience, and allows us to release all of our models to Android.

  • Model training update (Evgeny Pavlov, Jaume Zaragoza): 

    • Retrained old models for 11 high-resource languages with the highest impact on users. Now they have better quality, are more robust and translate short phrases better (German, Spanish, French, Portuguese, Italian, Dutch, Finnish, Estonian, Czech, Bulgarian, Polish) (translations/1239) 

    • Trained Icelandic

    • Improved English to Ukrainian and Catalan

    • Improved English to Chinese Simplified and Japanese by fine-tuning on LLM-generated high-quality datatasets

    • Redistilled some models from base to base-memory architecture to reduce memory consumption (Arabic, Chinese, Japanese, Korean)

    • Implented a new metric to measure how well models translate short phrases

    • Experimented with document-level translations as a hackathon on MT Marathon 2025 in Helsinki

    • Presented LLM data generation approach on MT Marathon

  • (July 2025) Language Release (All Platforms, All Features)

    • fa → en (Translations from Persian)

    • gu → en (Translations from Gujarati)

    • he → en (Translations from Hebrew)

    • hi → en (Translations from Hindi)

    • kn → en (Translations from Kannada)

    • ml → en (Translations from Malayalam)

    • ms → en (Translations from Malay)

    • sq → en (Translations from Albanian)

    • te → en (Translations from Telugu)

    • en → bn (Translations into Bangla)

    • en → fa (Translations into Persian)

    • en → gu (Translations into Gujarati)

    • en → he (Translations into Hebrew)

    • en → hi (Translations into Hindi)

    • en → kn (Translations into Kannada)

    • en → ml (Translations into Malayalam)

    • en → ms (Translations into Malay)

    • en → ta (Translations into Tamil)

    • en → te (Translations into Telugu)

  • (July 2025) Language Release (Android)

    • ar → en (Translations from Arabic)

    • ja → en (Translations from Japanese)

    • ko → en (Translations from Korean)

    • ru → en (Translations from Russian)

    • zh → en (Translations from Simplified Chinese)

    • en → ar (Translations into Arabic)

    • en → ja (Translations into Japanese)

    • en → ko (Translations into Korean)

    • en → ru (Translations into Russian)

    • en → zh (Translations into Simplified Chinese)

  • (Sept. 2025) Language Release (All Platforms)

    • bn → en (Translations from Bangla)

    • is → en (Translations from Icelandic)

    • en → az (Translations into Azerbaijani)

    • en → bn (Translations into Bengali)

    • en → is (Translations into Icelandic)

Phabricator , moz-phab, and Lando

Release Engineering and Release Management

  • The Release Management team shipped 3 major desktop releases (Fx141, Fx142, & Fx143), 7 dot releases, and also rolled out 9 ESR releases.

  • The Release Management team have been facilitating the transition to a weekly release cycle for iOS by coordinating timelines, updating processes, and supporting engineering/QA teams throughout the migration. This work is culminating with the v143.2 release set to launch as the first “true” weekly Firefox iOS release.

  • Pascal Chevrel catalogued and updated/added/removed all the third-party dictionaries we ship in Firefox Desktop per locale — part of a broader effort to address regional usability papercuts that affect the attractiveness of Firefox for non-English speakers. You can browse the full inventory here: https://vanilla.pascalc.net/dictionaries/ 

  • Dianna Smith updated the Release Calendar for 2026, which includes longer development cycles during the holiday seasons.

  • Pascal Chevrel updated whattrainisitnow.com with our 2026 calendar forecast, the ESR 115 support extension until March 2026, better mobile support, new API endpoints to help the security team automate the generation of security advisories and integrated feedback from PMs and developers.

  • Dianna Smith and Ryan VanderMeulen have been working on getting Firefox listed in the Huawei store. It is now available here: https://appgallery.huawei.com/app/C115133357 

  • Donal Meehan has been working on automating previously-manual steps to application-services and firefox-ios repos when new releases are cut from main.

  • Ryan VanderMeulen worked with the Android team to finalize, announce, and execute plans for dropping support for Android versions <8 and the 32-bit x86 architecture. Additionally, he helped shepherd support for 16KB page sizes needed for Google Play Store policy compliance.

  • Ryan VanderMeulen added universal APK builds for Fenix to our build system in response to a partner request.

  • Julien Cristau worked on planning and execution of the end of support for Firefox on 32-bit x86 Linux.

Mozregression 

  • Zeid released mozregression 7.0.0, which includes bug fixes and upgrades, removes support for Python 3.8, adds support for Python 3.13. It also included some GUI build fixes and updates for Ubuntu binaries.

Other

  • MFBT modernization, using more standard constructs, obsoleting mozilla/TemplateLib.h  (1986593) , MOZ_NORETURN (1990641), mozilla::MulOverflowMask (1990262), MOZ_ALIGNED_DECL (1990249), MOZ_MAYBE_UNUSED and ATTRIBUTE_UNUSED (1981331)


Thanks for reading and see you next month!


Reply all
Reply to author
Forward
0 new messages