PSA: Fallback mechanism for virtual test expectations

[Xianzhu Wang]

Hi,

Note that here "expectations" means the entries in web_tests/TestExpectations, etc., instead of baseline files (-expected.*). We already have fallback mechanism for virtual test baselines.

Now we no longer need duplicated expectation entries for the base test and virtual variants in web_tests/TestExpectations etc. if their expectations are the same, like:

crbug.com/123456 foo/bar.html [ Pass Failure Timeout ]

crbug.com/123456 virtual/suite1/foo/bar.html [ Pass Failure Timeout ]

crbug.com/123456 virtual/suite2/foo/bar.html [ Pass Failure Timeout ]

If a virtual test has different expectations, the following entry will override *all* base expectations:

virtual/suite/foo/bar.html [ Pass ]

A special rule for [ Slow ] (which is basically transparent to most developers) is that it's always inherited and merged. This is because [ Slow ] expectations are in a separate file SlowTests, and the situation would be composited if we allowed overriding of [ Slow ].

We haven't enabled this fallback mechanism for [ Skip ] yet because of concern about unexpected skips of virtual tests. If you are interested, you can look at this CL and comment. <rant>We are using too many Skips where we should not use it.</rant>

With this fallback mechanism, we removed about 500 lines of redundant expectations. Will remove more when we enable it for [ Skip ].

BTW, we also made expectations in FlagExpectations/* override (instead of merge with) main expectations. Previously we didn't have a way to expect a test to always pass in a flag expectation file.

References:

https://bugs.chromium.org/p/chromium/issues/detail?id=1072015

https://chromium-review.googlesource.com/c/chromium/src/+/2173656

https://chromium-review.googlesource.com/c/chromium/src/+/2178172

https://chromium-review.googlesource.com/c/chromium/src/+/2178173

Xianzhu

[Xianzhu Wang]

Since crrev.com/766598, we also let virtual tests inherit [ Skip ] if not overridden.

Note that sometimes we intentionally skip some base tests which are only meaningful to run with specific flags in virtual tests. In these cases we need to override the [ Skip ] with [ Pass ], like:

In NeverFixTests:

# These tests run under virtual suites only.

crbug.com/123456 foo/* [ Skip ]

virtual/suite1/foo*/ [ Pass ]

virtual/suite2/foo*/ [ Pass ]

[Philip Rogers]

Thank you Xianzhu! This will reduce churn and flakiness because it was easy for sheriffs to forget to add virtual expectations. And -700 lines of expectations!

[Chris Harrelson]

+1, awesome work!

On Thu, May 7, 2020 at 4:48 PM Philip Rogers <p...@chromium.org> wrote:

Thank you Xianzhu! This will reduce churn and flakiness because it was easy for sheriffs to forget to add virtual expectations. And -700 lines of expectations!

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAJgFLLsvUStOM%2BZJ7sGTPc_X1CzH4xZKM1frgHPOnfD%3D7vM_HQ%40mail.gmail.com.