run_web_tests.py --additional-driver-flag no longer matches flag-specific expectations or baselines

22 views
Skip to first unread message

Xianzhu Wang

unread,
Aug 10, 2022, 2:16:16 PM8/10/22
to blink-dev
Hi,

After this CL, run_web_tests.py --additional-driver-flag no longer matches additional expectations under web_tests/FlagExpectations or additional baselines under web_tests/flag-specific. Please use --flag-specific instead if you need that behavior.

The reason for the change is that the expectations file / baseline directory matching rule is complicated and sometimes problematic when --additional-driver-flag specifies multiple flags. --flag-specific can avoid the problems. Some new tools, e.g. remove_stale_expectations.py, support --flag-specific only.

Note virtual suites are more suitable for testing small sets of web tests with flags. See this doc for more details. Also note that for a blink runtime enabled feature that just adds an API without skipping any existing code path, setting status: "test" or status: "experimental" in runtime_enabled_features.json5 is preferred to a virtual suite or a flag-specific configuration.

Xianzhu
Reply all
Reply to author
Forward
0 new messages