Looking for input/guidance on maintenance of test coverage during long rollout of upcoming feature

[Colin Blundell]

Hi Chrome infra folks,

The Viz team is readying experimenting with the TreesInViz feature on Dev. This is a feature that broadly changes how the compositor operates, and one that we expect to take up to a year if not more to roll out fully.

As part of this, we're preparing to turn this feature on in fieldtrial_testing_config.json. However, this presents us with a tricky question:

We need to start getting CI coverage of the new codepaths as they will continually regress otherwise, but given the broad impact of the feature and the length of time that it will take to roll out, we don't want to lose test coverage of the existing production codepath either. Specifically, layout tests and browsertests meaningfully exercise different codepaths based on whether the feature is enabled.

Do you have any advice as to whether there are previous examples/best practices/feasible approaches to maintaining CI coverage across both codepaths?

Thanks,

Colin

[Andrew Grieve]

@Henrique Nakashima just set up android-10-x86-nofieldtrial-rel to run without fieldtrail_testing_config.json (bug 447426928). It runs android_browsertests, but if you want desktop coverage as well, you could see about changing an existing CQ bot for a no-field-trial variant (that's what Henrique did), or add a new CI bot.

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAMGE5NEEAAU3-VSiyidLQTWAgGh_%2BRSdxxapRw_xoyM_%2B-6u1g%40mail.gmail.com.

[Colin Blundell]

Thanks, Andrew! As Henrique is OOO: Do you know who he engaged with to determine the plan and work through the execution of it?

[Colin Blundell]

On Tue, Oct 7, 2025 at 3:34 PM Colin Blundell <blun...@chromium.org> wrote:

Thanks, Andrew! As Henrique is OOO: Do you know who he engaged with to determine the plan and work through the execution of it?

+Andrew Grieve Friendly ping on ^, thanks!

[Colin Blundell]

On Thu, Oct 9, 2025 at 2:36 PM Colin Blundell <blun...@google.com> wrote:

On Tue, Oct 7, 2025 at 3:34 PM Colin Blundell <blun...@chromium.org> wrote:

Thanks, Andrew! As Henrique is OOO: Do you know who he engaged with to determine the plan and work through the execution of it?

+Henrique Nakashima Could you give insight on the above question? Thanks!

[Ben Pastene]

If you want additional CI/post-submit coverage, that's mostly self-service these days via https://chromium.googlesource.com/chromium/src/+/main/docs/infra/new_builder.md (currently being revamped in https://crrev.com/c/6907183).

If you want additional CQ/pre-submit coverage (or to change pre-existing CQ/pre-submit coverage), the implementation of the change is self-service. But policy wise, you should loop-in the browser infra team: http://team/1283282796787 to review and get their sign-off. There's also a GWSQ alias for the team via chrome-dev-infra-reviewers@.

If you're asking more about the automated testing strategy of the feature over ~1-year rollout period, the infra team can help advise. You can kick off a discussion about it on http://g/chrome-dev-infra. But we won't be able to prescriptively say what's required for the rollout, that's more up to the feature owners since they have the best context on what kind of regressions can be expected on or off the experiment. The CQ/presubmit is curated as a selection of builds/tests that catch as many regressions as possible while not consuming too many machine resources. So it's a constant, delicate balancing act.

A note about fieldtrial_testing_config.json: I believe non-branded builds use fieldtrial_testing_config.json, while Chrome-branded builds do not. All of Chrome's CQ tests are non-branded builds, while ~half of Chrome's CI tests are branded builds. So doing nothing but adding entries to fieldtrial_testing_config.json would affect everything on the CQ + half of CI.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAMGE5NH6bgaDKqj4AUAgeT22TanAt7RHKuqs10c%2BJQAkv0gASw%40mail.gmail.com.

[Henrique Nakashima]

Hi Colin,

The discussion that led us to create android-10-x86-nofieldtrial-rel is mostly in crbug.com/422758891. I share your concerns about losing coverage of code paths not reached without fieldtrial_testing_config.json.

We're in the process of moving android-10-x86-nofieldtrial-rel to CQ. It's currently green and gardened, but we have to move around render tests from android-x86-rel to android-x64-rel before making the switch.

In this bot, fieldtrial_testing_config.json is only disabled in Java test apks (see chrome.android.star).

I think it makes sense to enable it for android_browsertests and content_browsertests too, however a lot (4000 initially) of failures in content_browsertests will need to be handled. We have this run because that was briefly what was happening, but I reverted the change to non-Java tests in this CL to control the scope of the task. In desktop, I imagine you'd need to handle a similar number of failures.

[Henrique Nakashima]

I spot checked those 4000 failures and most of them are because of a subset of failures which cause a timeout before the test even runs, so it's really an unknown amount of work.

[Colin Blundell]

Thanks, Ben and Henrique!

On Tue, Oct 14, 2025 at 10:57 PM Henrique Nakashima <hnaka...@google.com> wrote:

I spot checked those 4000 failures and most of them are because of a subset of failures which cause a timeout before the test even runs, so it's really an unknown amount of work.

^ This is sobering information, but I'm trying to reconcile it with Ben's point that "half the CI builders aren't using fieldtrial_testing_config.json". 

+Ben Pastene I guess none of those CI builders are running android_browsertests or content_browsertests on Android?

[Colin Blundell]

On Wed, Oct 15, 2025 at 2:13 PM Colin Blundell <blun...@chromium.org> wrote:

Thanks, Ben and Henrique!

On Tue, Oct 14, 2025 at 10:57 PM Henrique Nakashima <hnaka...@google.com> wrote:

I spot checked those 4000 failures and most of them are because of a subset of failures which cause a timeout before the test even runs, so it's really an unknown amount of work.

^ This is sobering information, but I'm trying to reconcile it with Ben's point that "half the CI builders aren't using fieldtrial_testing_config.json". 

+Henrique Nakashima, +Vasiliy Telezhnikov brought up the excellent point that the failures here are almost certainly due to the fact that the Android emulator bots require the passthrough decoder in order to work around lack of GLES3 support, and the passthrough decoder is currently turned on via fieldtrial_testing_config.json. You might want to turn that on explicitly and rerun.

WRT our plan for TreesInViz here: While the idea of a desktop no_fieldtrial_testing_config bot is compelling in general, it seems like an unknown amount of effort and I don't want to couple moving forward on TreesInViz to it. In discussion with Mo and Victor yesterday we came up with the following approach: Turning on TreesInViz on a subset of platforms via fieldtrial_testing_config.json (e.g. Windows and Android). This will preserve coverage of cross-platform codepaths across both configurations during the rollout. I expect interaction of TreesInViz with platform-specific codepaths to be limited - it does exist (e.g., we've seen that TreesInViz interacts with not-yet-shipped fluent overlay scrollbars, which are Windows-only), but this overall seems like the best balance.

Feedback welcome, of course!

Best,

Colin