Enabling Experimental Regression Test Selection (RTS) on CQ Dry Runs

94 views
Skip to first unread message

Struan Shrimpton

unread,
Jan 27, 2023, 3:34:21 PMJan 27
to chromi...@chromium.org, Gary Tong, Erik Staab

Hi chromium-dev,


We are planning to enable Regression Test Selection (RTS) on 10% of all CLs that go through CQ for the next two weeks (2023-01-30 to 2023-02-10). We are gathering data that will help us decide if and how to enable RTS on all dry runs. If your CL is in the experiment, your dry run will be faster, but on submission, we will run the remaining tests that the dry run skipped.


What is Regression Test Selection (RTS)?

RTS is a selection strategy to run a minimal set of tests to validate a CL by deciding which tests are the most important to run based on which files have changed. If you have tried quick run 🚀 you’ve already used RTS.


Why are we doing this?

RTS-enabled builds use far fewer resources to determine if a patchset is likely safe to commit. To detect 95% of failures we only use about 65% of the testing resources. After we apply the current model to dry run, improving the quality of the selection model has huge potential for even more savings. We can reapply these saved resources to improve the cycle times of CQ as a whole.


Will this lead to more reverts?

We still run all tests before landing a CL so there should not be an increase in reverts. A subset of tests is run during CQ+1 while the rest of the tests are run at CQ+2. If CQ+1 is not used CQ+2 will continue to run all the tests at the same time.


What if RTS causes problems with my CL?

If RTS needs to be disabled, add this footer to the commit message:


Disable-Rts:True


This will prevent the experiment from affecting the CL. If you believe the build failed because of RTS please file a bug using the 'Infra>Client>Chrome' component.


On behalf of Chrome Operations,

Struan


Olamide bobby

unread,
Jan 30, 2023, 4:29:53 PMJan 30
to Chromium-dev, ssh...@google.com, gat...@google.com, Erik Staab
Experiment

Struan Shrimpton

unread,
Feb 8, 2023, 2:00:58 PMFeb 8
to chromi...@chromium.org, Gary Tong, Erik Staab
Hi chromium-dev,

This experiment had an unexpected problem that needed to be resolved and caused a week delay in being enabled. The new timeline is 2023-02-06 to 2023-02-17.


On behalf of Chrome Operations,

Struan

Reply all
Reply to author
Forward
0 new messages