Announcing Chrome for Testing per-commit builds for enhanced bisecting

[Kuan Huang]

Hi Chromium Developers,

We're excited to announce the availability of Chrome for Testing (CfT) per-commit builds, a new resource designed to make bug bisection faster and more precise, especially for issues that are challenging to reproduce in internal environments.

Previously, CfT builds were aimed at use cases other than bisecting, and made available per released Chrome version. In addition to these per-version CfT builds, we now offer per-commit CfT builds.

Have you ever encountered a bug in Chrome and struggled to pinpoint the exact commit that introduced it?  For external and third-party developers, bisecting Chrome regressions can sometimes be difficult without access to readily available, publicly accessible builds at a granular commit level.

To address this, we've launched Chrome for Testing per-commit builds!  These builds are:

• Publicly accessible: Anyone can download and use them — no logins required!

• Per-commit granularity: We aim to provide builds as close to a per-commit level as possible, giving you finer-grained control for bisecting. While Mac builds are generally per-commit, please note that due to resource constraints, Windows and Linux builds might be archived slightly less frequently, potentially capturing a few commits per build.

• Built for bisecting: These builds are designed to be similar to branded Chrome, making them ideal for regression testing and identifying culprit commits.

How to use them with bisect-builds.py:

Integrating these new builds into your existing bug bisection workflow is incredibly easy. If you are already using bisect-builds.py (the recommended tool for bisecting Chrome regressions), simply add the -cft flag to your command line!

For example:

python tools/bisect-builds.py -cft -a <platform> -g <good-revision> -b <bad-revision> -- <flags-for-chrome>

Supported platforms:

Currently, per-commit CfT builds are available for the following platforms:

• macOS (mac-arm)

• Windows (win64)

• Linux (linux64)

Benefit from historical builds:

We've already built up a history of 3 months of per-commit CfT builds, and we'll store these historical builds for 1 year. This historical data will eventually expand to encompass a full year of history, allowing you to start bisecting regressions from recent history right away!

By using Chrome for Testing per-commit builds, you can:

• More efficiently identify regression culprits: Pinpoint the problematic commit with greater accuracy.

• Contribute more effectively to Chromium bug fixing: Provide precise information to Chromium developers, speeding up issue resolution.

• Simplify your bug investigation process: Leverage publicly available builds for easier and faster bisection.

We encourage you to try out Chrome for Testing per-commit builds for your next bug bisection task. Please let us know if you have any questions or feedback!

Happy bisecting!

The Chrome EngProd & Browser Automation teams

[Rick Byers]

Thank you Kuan, Mathias and others involved in this. This is awesome news and great work!

With my Chrome ATL hat on I can say that per-revision bisects have been one of the most powerful tools we've ever deployed for avoiding and mitigating regressions. No amount of testing and breaking change process rigour can completely prevent all breaking changes at the scale of the whole web. The #1 lesson I've learned from nearly a decade of trying to reduce the pain of Chrome breaking changes, is that fast and reliable root-cause analysis is the key to keeping browser product quality up without completely tanking engineering velocity.

Every Chrome release we tend to get some reports of changes in website behavior, and we now have a pretty well-oiled machine for reliably getting those bug reports prioritized and routed to the responsible engineer and (where appropriate) reverting or disabling the change, often all in under 24 hours. Per-revision bisects have meant the difference between release managers confidently routing P0 bugs and reverting CLs, vs. looking for volunteer engineers who can help with difficult debugging and guesswork at which of ~30 CLs is most likely the cause of a change. But for years this system has depended on Google's internal per-revision build archive, which means it falls down when we get a report from a developer of an issue that can't be reproduced on a public webpage.

With this announcement, the entire web developer community now has the same power that Google has to easily (without doing their own Chromium builds) blame any change in behavior on Chrome on the CL that caused it, and by extension help ensure quick reverts of web-breaking CLs which are lacking a full breaking change analysis and approval. While I think we're pretty good at responding to bug reports of most regressions, I am thrilled to have this capacity more broadly available and hope that it helps empower more stakeholders in the process of ensuring the high quality of Chromium releases!

Rick

[Philipp Hancke]

This is indeed great news! https://groups.google.com/a/chromium.org/g/chromium-dev/c/abRPgin7Kgs/m/YjpcdJmhDQAJ

is a recent example where per-commit bisect would have been even more helpful, from what I recall it was two possible culprit changes. Great reaction from the Chromium (and libxml upstream) side nonetheless.

Out of curiosity, can this be extended to "rolls" for affiliated third-party libraries like WebRTC?

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAFUtAY8HDp4cwc_PomJBQ8-6Ub0D2txtNQY7AdWtfuh6c90oBg%40mail.gmail.com.

[guest271314]

+1. I've been just using Chromium Developer build recently.

Would be great to be able to *exclude* stuff like Google Safe Browsing that I disable immediately, and screen_ai which on Linux is ~100 MB that I have no use for. A basic checkbox form. Then build. Just an idea.

[Harald Alvestrand]

WebRTC rolls are already Chromium commits, so you can pinpoint the roll, but not the WebRTC commit, I think.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CADxkKiKCu4Mwt7Ss5M41LoV5xT%2B38QVTjJHvRUQv9TvE-oSR-g%40mail.gmail.com.