WPT flake caused by Chrome's butterbar

[Ali Juma]

I'm working on upstreaming web platform tests for IntersectionObserver, and am running into flakiness caused by the drawing of the "Chrome is being controlled by automated test software" butterbar that appears when using chromedriver. The tests depend on the height of the window (they grab this using document.clientHeight when the test begins). This changes when the butterbar is drawn, so if this happens after the tests have started running, we get failures (where the results are off by 39 pixels in the y-dimension, and this appears to be the height of the butterbar). Delaying the start of the test using triple-raf seems to "work" but even that seems inherently fragile, since the butterbar is initiated by the browser process so I'm not sure we can depend on the height changing after any constant number of renderer frames.

The tests aren't flaky when run as layout tests (which doesn't involve chromedriver afaik), only when run using WPT tooling.

Any ideas about the best way to proceed? Is there a way to turn off the butterbar, or some reliable way to wait for it (that's still reasonable to add to a web platform test)?

Thanks,

Ali

[Philip Jägenstedt]

+John Chen who is working on ChromeDriver.

I didn't know about this bit of UI. How are you running wpt to encounter this problem?

Looks like it's added in https://cs.chromium.org/chromium/src/chrome/browser/ui/startup/startup_browser_creator_impl.cc?type=cs&l=826 and that there's no second command line arg to disable it. I would dig through the history of that to see why it was added, and if a way of disabling it would be compatible with the original motivations.

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CANLC6v0e_EwoSPyu4xTYK2fziPhrVBi5UzuLKR2LkoxeGKgwhQ%40mail.gmail.com.

[John Chen]

Ali,

If you're using ChromeDriver 2.29 or above, there are two experimental options that you can use to disable this UI. In Python, the code would be like (code in other languages are similar):

options = webdriver.ChromeOptions()

options.add_experimental_option('useAutomationExtension', False)

options.add_experimental_option('excludeSwitches', ['enable-automation'])

driver = webdriver.Chrome(chrome_options=options)

This prevents ChromeDriver from adding --enable-automation switch to Chrome command line, thus avoiding the UI to indicate using of automation extension. This will prevent some of the less commonly used feature from working, for example, you won't be able to launch Chrome applications through ChromeDriver without the automation extension. On older versions of Chrome (58 and below) you also won't be able to resize the Chrome window.

Thanks,

John

On Wed, Jun 14, 2017 at 5:12 AM, Philip Jägenstedt <foo...@chromium.org> wrote:

+John Chen who is working on ChromeDriver.

I didn't know about this bit of UI. How are you running wpt to encounter this problem?

Looks like it's added in https://cs.chromium.org/chromium/src/chrome/browser/ui/startup/startup_browser_creator_impl.cc?type=cs&l=826 and that there's no second command line arg to disable it. I would dig through the history of that to see why it was added, and if a way of disabling it would be compatible with the original motivations.

On Tue, Jun 13, 2017 at 12:04 AM Ali Juma <aj...@chromium.org> wrote:

I'm working on upstreaming web platform tests for IntersectionObserver, and am running into flakiness caused by the drawing of the "Chrome is being controlled by automated test software" butterbar that appears when using chromedriver. The tests depend on the height of the window (they grab this using document.clientHeight when the test begins). This changes when the butterbar is drawn, so if this happens after the tests have started running, we get failures (where the results are off by 39 pixels in the y-dimension, and this appears to be the height of the butterbar). Delaying the start of the test using triple-raf seems to "work" but even that seems inherently fragile, since the butterbar is initiated by the browser process so I'm not sure we can depend on the height changing after any constant number of renderer frames.

The tests aren't flaky when run as layout tests (which doesn't involve chromedriver afaik), only when run using WPT tooling.

Any ideas about the best way to proceed? Is there a way to turn off the butterbar, or some reliable way to wait for it (that's still reasonable to add to a web platform test)?

Thanks,

Ali

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsub...@chromium.org.
To post to this group, send email to platform-predictability@chromium.org.

[Ali Juma]

Thanks John, using those options fixes the flakiness!

Philip, I'm running wpt using "python check_stability.py chrome".

The next thing I need to figure out is whether those options break any existing tests. What's the best way to do a before/after comparison for such a change? Is there a script that does something like that?

Thanks,

Ali

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CANLC6v0e_EwoSPyu4xTYK2fziPhrVBi5UzuLKR2LkoxeGKgwhQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAN4pA0e92am_wDWAY510nHCi_Nn4u_HwHnGFAxyW7A-hQ6HY9w%40mail.gmail.com.

[Ali Juma]

One more question: I notice that using wptrun sets things up a bit differently than "python check_stability.py chrome". Specifically, wptrun calls set_window_size to get a 600x600 window, but check_stability seems to just use whatever window size Chrome happens to be. Is wptrun the officially supported way (so should check_stability really be matching wptrun's behavior)?

Thanks,

Ali

[Philip Jägenstedt]

I don't believe there's a pre-packaged way to compare two runs, but in https://bugs.chromium.org/p/chromium/issues/detail?id=651572 I did something similar when comparing runs of Chrome, Edge and Firefox. +James Graham or +Jeff Carpenter might be able to give you some clues about how to get the results into a database in the first place.

As for window size, I've seen some issues around CSS where +Geoffrey Sneddon has been talking about assumptions that test make. And I think that +Bob Holt has been thinking about perhaps merging check_stability.py into wptrun, and I think any differences like that would be a good idea to remove.

[Jeff Carpenter]

I've been using wptrunner to run the tests. I have a WIP PR for a wptrunner log summarization script, however it only provides a pass/fail number at the test file level (not for individual subtests). That script could be a good starting point for getting and comparing more detailed subtest results (which is also on my roadmap at some point).

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAARdPYfdL6%2BEbv1fUMoJRaiusn22Z%2B%2Bc1uPObF53bzt4qY%2BNKA%40mail.gmail.com.

[Ali Juma]

Thanks everyone!

Adding the experimental options suggested earlier prevents window resizing on Chrome 59 (it works fine on 61 though), and this used by reftests. So for now my plan is to add these flags only for testharness tests. Running locally, I didn't see any new failures that seemed to be triggered by these flags.