PSA: Demoting "not_site_per_process_blink_wpt_tests" on "linux-rel" to CI-only

[Jeffrey Yu]

Hi Chrome devs,

We'll be demoting "not_site_per_process_blink_wpt_tests" running on the "linux-rel" builder from CQ to run on CI only.

As previously announced, the Chrome Dev Infrastructure team is currently addressing a capacity issue on the CQ builders that's resulting in high pending times and slow CQ runs across the board.

The team has tools that analyzed each test suite across the previous year of CL history, and determined whether those test suites provide additional value in detecting failures or bugs. The Regression Test Selection (RTS) Suite Analysis Tool analyzes CQ test runs and calculates the Recall rate for every test suite, the Recall rate being the percentage of failed CLs in the given time period that would still have failed if the test suite was not run on the builder (successful CLs do not factor into this number). 

• FAQ and more detailed explanation here: Demoting High-Recall Tests From CQ

To free up CQ capacity, the team will phase out certain resource-expensive test suites, demoting them to run on CI only. "not_site_per_process_blink_wpt_tests" was analyzed through the RTS Suite Analysis Tool and through SQL analysis; even if demoted from CQ, no additional CLs would have made it through CQ. The team has previously made similar changes with good results and no regression in test coverage. 

"not_site_per_process_blink_wpt_tests" will continue to run on CI, as well as on other CQ builders, including:

• android-x86-rel

• linux-chromeos-rel

• mac-rel

• win-rel

The CI will be monitored to verify no instances of it failing in isolation, but after this change, if you suspect a CL was reverted because of this demotion, please file a bug (using this template) and contact us.

If you have any questions or concerns, please let us know!

[Ben Pastene]

To correct an inaccuracy in this PSA: "not_site_per_process_blink_wpt_tests" will NOT run on ANY default CQ bot after this removal. So it will NOT run on android-x86-rel, linux-chromeos-rel, mac-rel, or win-rel, as previously claimed.

Apologies to all for the confusion regarding these proposals. We'll try to convey things more clearly going forward.

And I know removing CQ coverage can be worrying. But I believe we stand by the analysis here: there hasn't been a single CL in the past 6 months where this suite was the only failure. Therefore it's not providing any valuable coverage that other CQ suites don't also provide. We can always restore it if things change.

Please reach out to me if you have any questions or concerns.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAD4jpnN_-62FdJP%3DFAmPz-TYRPgAs7FukJuTB_7efa5qRSkWQA%40mail.gmail.com.