Re: New run_web_tests.py feature for WPT - fuzzy reftests

[Chris Harrelson]

Nice!

Rendering team: can someone volunteer to try this out on a test they couldn't upstream to WPT or is marked as failing because of fuzzy diff?

On Mon, Dec 7, 2020 at 4:58 PM 'Stephen McGruer' via rendering-core-dev <rendering...@chromium.org> wrote:

Hi rendering-core!

I wanted to publicize a newly supported feature in run_web_tests.py for WPT tests - fuzzy matching for reftests (or 'fuzzy reftests' for short). This feature is enabled by including a meta tag in the reftest (example):

<meta name=fuzzy content="maxDifference=0-15;totalPixels=0-300">

The 'maxDifference' and 'totalPixels' variables define the maximum difference in any of the RGB channels and the total number of pixels that are allowed to differ respectively. They are a bit complicated, so check out the full documentation for details. The variable names can be omitted for conciseness - the above is equivalent to:

<meta name=fuzzy content="0-15;0-300">

One Chromium-specific tip; if you pass --debug-rwt-logging to run_web_tests.py, you will be able to see the output lines from the underlying image_diff.cc code. This is helpful when trying to determine the fuzzy bounds for a test, e.g.:

./third_party/blink/tools/run_web_tests.py --debug-rwt-logging external/wpt/css/css-images/image-orientation/image-orientation-none-content-images.html

...

19:56:32.029 1717778 "/usr/local/google/home/smcgruer/chromium/src/out/Release/image_diff --diff /tmp/tmpY0o691/actual.png /tmp/tmpY0o691/expected.png /tmp/tmpY0o691/diff.png --fuzzy-max-channel-diff=0-5 --fuzzy-max-pixels-diff=0-150" took 0.05s
19:56:32.030 1717778 Found pixels_different: 121, max_channel_diff: 3
Allowed pixels different; 0-150, channel diff: 0-5

We hope this feature will be useful to y'all, in enabling more testing of features that would be otherwise affected by subpixel shift or AA. Feedback and questions are most welcome. I personally think there's a lot of scope to improve fuzzy reftests, but we're starting with at least supporting them :)

Thanks,

Stephen

--

smcgruer • he / him

--
You received this message because you are subscribed to the Google Groups "rendering-core-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rendering-core-...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/rendering-core-dev/CADY3MaejipBpDnPz9H6nmiLWGpF6xTeRi0XZZkRqnZpzVwgkgg%40mail.gmail.com.

[Stephen Chenney]

I'm not aware of cases where we have not upstreamed tests for this reason. In general we were upstreaming them and marking them as failing. Stephen McG has already been working through the tests in TestExpectations marked as failing for this reason.

Of note there are several non-wpt tests that fail due to anti-aliasing and we would do well to convert those to WPT and make use of the feature. Once upon a time I commented on such tests in the expectations file but other cases have probably accumulated.

Cheers,

Stephen.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/rendering-core-dev/CA%2BN6QZv0rvvg%2Bm2bKfT2%2Bm1Z%3Drn%2BE%2BKmz3qRdHWuXFYckSnefw%40mail.gmail.com.

[Xianzhu Wang]

I remember we have occasionally modified some important non-wpt ref tests to pixel tests to workaround the subpixel mismatch issue. Will try to find some.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/rendering-core-dev/CAGsbWzQAQ8gEpNVBOF4C-g0-XkjP3a5tashTBJs7bkNrL4uByQ%40mail.gmail.com.

[Mason Freed]

Nice work!! That's great news. I have several tracking bugs from the backdrop-filter implementation (e.g. crbug.com/978481 and crbug.com/1035194) for tests that should now be able to be upstreamed to WPT. In several of those cases, I actually wrote the WPT and linked it to the bug, before rolling it back and making it a web_test.

Thanks,

Mason

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/rendering-core-dev/CADBxridzBWo%2BKcx88XjSsAb4vJDGQVZnYN1j3bjSrkZw8kEBYw%40mail.gmail.com.