Engineering Effectiveness Newsletter (Q2 2026 Edition)

8 views
Skip to first unread message

Andrew Halberstadt

unread,
Jul 9, 2026, 12:30:39 PM (13 hours ago) Jul 9
to dev-pl...@mozilla.org, Firefox Dev

Welcome to the Q2 edition of the Engineering Effectiveness Newsletter! The Engineering Effectiveness org makes it easy to develop, test and release Mozilla software at scale. See below for some highlights, then read on for more detailed info!

Highlights image.png

Detailed Project Updates

AI for Development image.png

  • Suhaib Mujahid deployed the initial version of Hackbot, a platform for building and running AI agents to automate parts of the Firefox development workflow.

  • Evgeny Pavlov ported the “Build Repair Agent” to Hackbot and deployed it for testing. It now monitors Firefox build failures and triggers the agent. When an analysis and a proposed patch are ready developers can be notified by email.

Bugzilla image.png

  • David Lawrence added a new GitHubPullRequests extension that renders a live status panel in the bug modal for any attachment whose content type is text/x-github-pull-request. A new REST endpoint fetches PR metadata (state, author, labels, latest review per reviewer) from the GitHub REST API on demand, and a client-side script populates a table with a "show closed/merged" toggle.
    image.png

  • Xavier L’Hour improved the user experience for developers, adding shortcuts to buglist.cgi for all, open, or closed bugs (1764713)

  • Xavier L’Hour added a new shortcut button to the bug page that allows users to quickly move spam bugs to the Invalid Bugs product (1684509).

Build System and Mach Environment image.png

  • Alex Hochheiden has been moving build system logic out of make to pave the way for a new build system backend (coming soon). See Bug 2038789.

  • Alex Hochheiden landed a 30%-50% (platform dependent) speedup for mach startup. See Bug 1775197.

  • Alex Hochheiden sped up subsequent configure runs by ~10s by adding caching to the mach taskgraph toolchain step. See Bug 2017746.

  • Alex Hochheiden reduced mach test startup overhead on Windows by 75%. See Bug 2018327.

  • Alex Hochheiden has achieved significant code deduplication and simplification by consolidating the Android Gradle configuration into convention plugins. There were also various Gradle configure-cache improvements. See Bug 2007013, Bug 1950099, Bug 2013417, Bug 2017752, and Bug 2017753.

Firefox-CI image.png

  • Julien Cristau added support for interactive tasks (aka one click loaners) on Windows and macOS

  • Andrew Halberstadt implemented mach try support with Github, being used in mozilla/enterprise-firefox-try and coming to Firefox soon.

  • Andrew Halberstadt implemented the machinery to start making Gecko CI tasks clone from Github.

  • Ryan Curran brought Firefox CI's Apple Silicon VM infrastructure into production. Building on the MacOS CI image pipeline established last year, he migrated test suites onto virtual machines and grew the macosx1500-aarch64-vms pool so Taskcluster now routes eligible jobs to VMs alongside physical hardware. This reduces reliance on physical Macs, increases CI capacity, and supports the ongoing migration off of older Intel-based macOS infrastructure

  • Jonathan Moss migrated Firefox CI's cloud-based Windows testing from Windows 11 24H2 to 25H2, moving the bulk of Firefox's Windows test coverage to Microsoft's latest platform and keeping CI aligned with the Windows version most commonly used by Firefox Desktop users

  • Florian Quèze created many dashboards to help dig into Mochitests and XPCShell tests

  • Ryan VanderMeulen landed a set of improvements to mach try chooser. The update adds an exclude filter, a clearer preview pane with removable job rows, an artifact-builds toggle, and a warning when a selection exceeds task-prioritization thresholds. It also fixes a bug where choosing Firefox for Android jobs would unintentionally clear selections for other platforms.

Lint, Static Analysis and Code Coverage image.png

  • Valentin Rigal and Bastien Abadie created a Code Review Bot prototype for publication of review comments using various source linters on Github

  • Morgan Rae Reschenberg added support for accessibility review to Code Review Bot

Mozregressionimage.png

  • Zeid fixed a bug in mozregression-gui on macOS, where the camera and microphone capture request was getting rejected (released in 7.3.0). Thanks to bug report + tip from Andreas Pehrson.

PDF.js image.png


Phabricator image.png, moz-phab, and Lando image.png

  • Connor Sheehan improved the uplift experience by leveraging Lando to manage the assessment forms, train selection, and automatic application, so conflicts are detected earlier. The number of uplifts via Lando has out-paced those via Moz-Phab, and sailed through the rise in uplift numbers (likely due to more sec-bugs getting fixed and uplifted).

  • Zeid added support for private GitHub repositories in Lando, allowing security patches to be implemented in a private clone of a repo, and pushed to the public one.

  • Olivier Mehani finalized support for using the new Lando instance for try-pushes. This brings a host of QoL improvements which weren't backported to the old instance: better UTF-8 support, smarter conflict resolution and improved security and authentication. It is now processing about 1500 pushes / week (old Lando still processes about 50 / week).

  • Magnolia Liu implemented automatic pushes to Try for uplift requests, for faster feedback in case of issues.

  • Olivier Mehani added a view of a user's current and recent jobs on the landing page of Lando when authenticated.

  • Zeid identified and fixed the causes of some stability and reliability issues in Lando, which were causing increased downtime during deployments and on an ongoing basis.

  • Olivier Mehani deployed a PoC of reviewer selection on the GitHub pilot, allowing Herald-like mechanisms to GitHub PRs.

  • Olivier Mehani and Connor Sheehan (with Corey Bryant and Daniel Darnell) migrated the COMM project to GitHub https://github.com/thunderbird/thunderbird-desktop, sharing Firefox's syncing model.

Release Management and Engineering image.png

  • Donal Meehan drove the Release Management team's move to a weekly scheduled dot release cadence for Desktop and Android, starting with Firefox 151. This allows us to deliver fixes and approved uplifts to users faster and more predictably. This change is expected to reduce unplanned releases, improve release flexibility, and create a more consistent release rhythm across teams.

  • Dianna Smith drove the update to the Release Management team's Desktop major release rollout process, starting with Firefox 152. Instead of throttling to 0% on day 2, it will remain at 25% rollout for two days before moving to 100%, unless any issues arise. This should help us collect uptake and stability signals earlier while still allowing time to catch problems before full rollout. 

  • Pascal Chevrel completed the update to the dictionaries shipped with Firefox Desktop. The update added eleven new dictionaries, covering Croatian, English (UK), Georgian, Persian, Slovenian, Tajik, Tamil, Tibetan, Turkish, Welsh, and Xhosa, and refreshed nine others. This expanded the number of locales with a built-in spellchecker from 30 to 41 beginning in Firefox 152. Special thanks to Francesco Lodolo, Bryan Olsson, and the localization community for reviewing the patches and helping assess the quality of the dictionaries.

  • Pascal Chevrel delivered a range of improvements to WhatTrainIsItNow, including expanded it to cover weekly dot releases and ESR planned dot releases, added new uplift views including a dot-release uplifts page and a beta uplift graph, and published new APIs that surface train-selection and uplift guidance inside Lando. He also made performance improvements and a steady stream of fixes across the site.

  • At Pwn2Own 2026, Firefox came through with no successful exploits, thanks to preparation across many teams and individuals. Within Release Management, Ryan VanderMeulen drove pre-event patch readiness and Dianna Smith coordinated the releases during the event, including the 150.0.3 dot release, which mitigated the root cause behind several of the contest entries.

  • Dianna Smith built out release-health monitoring and alerting in Bigeye, giving Release Management a growing set of automated alerts that surface data anomalies earlier to aid in release health and regression detection. To make the capability easy to extend, she also created a guide for other teams to add monitoring and alerts for the areas they know best. Teams that want an earlier signal on their own metrics are encouraged to use the guide and help grow the coverage.

Release Operations 🔧

  • Ryan Curran built Hangar, a live dashboard for monitoring Firefox CI's worker pools. It consolidates fleet data from several systems into one view, giving Release Operations a single place to check fleet health and catch problems such as missing or quarantined workers early.

  • Ryan Curran created the iOS version of BuildWatch, and Andrew Erickson ported it to Android. BuildWatch lets you monitor Firefox CI try pushes from your phone, including live per-platform build status, failure summaries, and one-tap retriggers. It uses only public APIs, so no VPN is required.

  • Andrew Erickson and Mark Cornmesser developed Fleetbench, a tool for benchmarking Firefox CI workers. It currently measures CPU and ADB/USB I/O performance, helping Release Operations identify slow or outlier hosts before they skew performance test results such as Speedometer and trigger noisy or false regressions.

  • Andrew Erickson built Pool Classifier, a web app for viewing per-worker success rates across Taskcluster worker pools. It classifies newly completed tasks every 15 minutes, giving Release Operations a continuously updated view of worker health and helping surface problematic workers proactively.

  • Andrew Erickson created Fleetroll, a command-line tool Release Operations uses to manage and monitor long-running Linux, macOS, and Windows hardware hosts in Firefox CI Taskcluster. It deploys Puppet branch overrides and Vault secrets, audits what is actually applied, and surfaces each host’s Puppet and Taskcluster state in a live dashboard.

  • Mark Cornmesser built out a set of new worker-metrics dashboards in Yardstick, giving Release Operations clearer real-time visibility into the health of the Firefox CI hardware fleet. These include Linux worker status, Windows worker CPU and disk metrics, and a Windows job pickup and wait timeline, with alerting on key thresholds. The full set lives in the FXCI Hardware Workers folder in Yardstick.

  • Jonathan Moss expanded cloud cost reporting in Looker, adding Azure support alongside the existing GCP data and a cloud-provider filter on the FXCI task overview dashboard. The team can now break down Firefox CI compute costs by cloud provider, making it easier to track and compare spend across Azure and GCP.

Taskcluster image.png

  • Yaraslau Kurmyza added Azure fast deprovision taskcluster#8790  and concurrency taskcluster#8815 to improve worker scanner performance. This shows ~2x-4x scan time improvements already.

  • Contributor nitishagar  and Yaraslau Kurmyza added patches taskcluster#8514,  taskcluster#8784  to support compression in Taskcluster services API and Yarik worked with Fastly to resolve broken brotli support on the WAF edge side. Now services transmit significantly less data.

  • Yarik added a dedicated service account to log with read only permissions webservices-infra#11197. This allows tc-logview to be used safely by untrusted agents inside containers with narrow short-lived access tokens.

  • Yarik published queue forecasting dashboard experiments that continuously collects task events and trains models to enable and improve predictions on a task level (how long will it run, and when will it start). With future plans including extending it to the whole task group (mach try)

Treeherder image.png

  • Florian Quèze added treeherder#9540 "Show task group profile" item to the push action menu

  • Cameron Dawson, juungo and moijes12 implemented various Treeherder API performance improvements

  • Heitor Neiva added Git branch labels to pushes in Treeherder

  • Andrew Halberstadt implemented the ability for Treeherder to display multiple Git branches at once, enabling support for “try like” repositories in Github

Version Control image.png

Other image.png

  • Sylvestre converted our documentation from reStructuredText to MyST flavored Markdown


Thanks for reading and see you next quarter!


Reply all
Reply to author
Forward
0 new messages