Sorry, "-expected.txt ... should be ... eventually be removed" in my previous email was not clear. I meant for each individual -expected.txt we should eventually remove it because we should fix the failure. The same logic applies to TestExpectations. At any time we may allow a certain number of failures but we should keep the number as small as possible.
I think we should prefer TestExpectations to -expected.txt for WPT tests because the entries in TestExpectations have associated bugs which track the fixing process, unless we find a better way to track the fixing of the failures in -expected.txt. -expected.txt files do have their values, e.g. for partially-passing tests we can discover regressions and progressions of individual sub tests, but they should be rare.
I think separating -expected.txt from the tests has the following benefits:
- It makes it clear to blink developers that the files are not a part of WPT.
- It simplifies the WPT export/import process and others by reducing blink-specific files under external/wpt.
It does make it more difficult to find -expected.txt, but we already have the similar well-known logic for platform-specific baselines. Though platform-specific baselines are rare, ignoring a platform baseline can still cause surprises.
- to better show -expected.txt for passing tests
- to show information about tests without actually running the tests
WDYT?