Running interactive ui tests on Linux with xvfb and icewm

[Chad Faragher]

I think that the testing and infrastructure page
(http://www.chromium.org/developers/testing) needs a note that, on
Linux, the interactive ui tests need to be run using xvfb and icewm.

When I first started looking at the interactive ui tests, I noticed
that they were extremely flaky. After much debugging I discovered
that my window manager would move the window around and this would
interfere with the ability of the test to find a window's size and
position, and correctly deliver a click into that window at the
correct point on the screen.

There are no visible warnings that the tests need to be run in an
isolated x server and window manager to get deterministic behavior.
oshima@ was the first to suggest to me that the tests actually NEED to
be run this way and are invalid if run outside that context.

I'd put up some instructions, but I'm not sure of the best way to set
up for a manual session. I was thinking something like this:

Session setup:
Xvfb :6.0 & DISPLAY=:6.0 icewm &

Running a test in that session:
DISPLAY=:6.0 out/Debug/interactive_ui_tests
--gtest_filter=MyTestClassName.MyTestMethodName

But petermayo@ suggested that I should use:
xterm -e '~/bin/Xvfb :6.0 & DISPLAY=:6.0 icewm'

And to peek at the display I should use use:
xwd -display :6.0 -root | xwud

Can anyone else comment on what the best instructions are for setting
up xvfb and icewm? And whether it's really actually necessary at all?

[Evan Martin]

On Fri, May 20, 2011 at 6:25 AM, Chad Faragher <wy...@chromium.org> wrote:
> I think that the testing and infrastructure page
> (http://www.chromium.org/developers/testing) needs a note that, on
> Linux, the interactive ui tests need to be run using xvfb and icewm.
>
> When I first started looking at the interactive ui tests, I noticed
> that they were extremely flaky.  After much debugging I discovered
> that my window manager would move the window around and this would
> interfere with the ability of the test to find a window's size and
> position, and correctly deliver a click into that window at the
> correct point on the screen.

I wasn't aware of this. It seems there are two reasonable approaches
to fixing it:
1) constraining the environment further, as you suggest below
2) fixing the tests to not be sensitive to this

Can you estimate how hard the latter is? In general relying on being
able to position the window directly is something that can fail on
Linux (as these tests are showing) so in some sense the testing
infrastructure is failing for the correct reason. (It depends on
whether we care about asserting that we control positioning
perfectly.)

> There are no visible warnings that the tests need to be run in an
> isolated x server and window manager to get deterministic behavior.
> oshima@ was the first to suggest to me that the tests actually NEED to
> be run this way and are invalid if run outside that context.

oshima should have updated the test running instructions!
(I personally haven't run interactive_ui_tests locally since... well,
as far back as I can remember.)

> I'd put up some instructions, but I'm not sure of the best way to set
> up for a manual session.  I was thinking something like this:
>
>  Session setup:
>  Xvfb :6.0 & DISPLAY=:6.0 icewm &
>
>  Running a test in that session:
>  DISPLAY=:6.0 out/Debug/interactive_ui_tests
> --gtest_filter=MyTestClassName.MyTestMethodName
>
> But petermayo@ suggested that I should use:
>  xterm -e '~/bin/Xvfb :6.0 & DISPLAY=:6.0 icewm'
>
> And to peek at the display I should use use:
>  xwd -display :6.0 -root | xwud
>
> Can anyone else comment on what the best instructions are for setting
> up xvfb and icewm?

If you do go down this route, the instructions you pasted above are
roughly the same commands.
I think it'd be better to use Xephyr though, because that will allow
you to see the test output (while still keeping it in a controlled
environment).

See the discussion under "Running tests" on the debugging page:
http://code.google.com/p/chromium/wiki/LinuxDebugging#Running_Tests
It links to more info about Xephyr, including the steps to set it up.

(...now that we use DumpRenderTree, I guess we should move the Xephyr
notes off of the layout tests page as it's no longer relevant to
layout tests in particualr.)

[Tony Chang]

On Fri, May 20, 2011 at 8:36 AM, Evan Martin <ev...@chromium.org> wrote:

On Fri, May 20, 2011 at 6:25 AM, Chad Faragher <wy...@chromium.org> wrote:
> I think that the testing and infrastructure page
> (http://www.chromium.org/developers/testing) needs a note that, on
> Linux, the interactive ui tests need to be run using xvfb and icewm.
>
> When I first started looking at the interactive ui tests, I noticed
> that they were extremely flaky.  After much debugging I discovered
> that my window manager would move the window around and this would
> interfere with the ability of the test to find a window's size and
> position, and correctly deliver a click into that window at the
> correct point on the screen.

I wasn't aware of this.  It seems there are two reasonable approaches
to fixing it:
1) constraining the environment further, as you suggest below
2) fixing the tests to not be sensitive to this

Can you estimate how hard the latter is?  In general relying on being
able to position the window directly is something that can fail on
Linux (as these tests are showing) so in some sense the testing
infrastructure is failing for the correct reason.  (It depends on
whether we care about asserting that we control positioning
perfectly.)

I think it's reasonable that some tests depend on being able to position the window directly.  For example, session restore tries to re-position the restored windows.

That said, these tests should still pass in metacity and compiz.  If you're using a tiling window manager (ion, awesome, xmonad, etc), it's better to use the Xvfb setup described.  The use of icewm on the bots is kind of arbitrary.  We could probably switch to metacity pretty easily.

 

> There are no visible warnings that the tests need to be run in an
> isolated x server and window manager to get deterministic behavior.
> oshima@ was the first to suggest to me that the tests actually NEED to
> be run this way and are invalid if run outside that context.

We should probably add a check to interactive_ui_tests to see if you're running metacity, compiz, or icewm.  If not, we could print a warning to stderr or something.

[Peter Mayo]

On Fri, May 20, 2011 at 11:36 AM, Evan Martin <ev...@chromium.org> wrote:

On Fri, May 20, 2011 at 6:25 AM, Chad Faragher <wy...@chromium.org> wrote: 

...

> I'd put up some instructions, but I'm not sure of the best way to set
> up for a manual session.  I was thinking something like this:
>
>  Session setup:
>  Xvfb :6.0 & DISPLAY=:6.0 icewm &
>
>  Running a test in that session:
>  DISPLAY=:6.0 out/Debug/interactive_ui_tests
> --gtest_filter=MyTestClassName.MyTestMethodName
>
> But petermayo@ suggested that I should use:
>  xterm -e '~/bin/Xvfb :6.0 & DISPLAY=:6.0 icewm'
>
> And to peek at the display I should use use:
>  xwd -display :6.0 -root | xwud
>
> Can anyone else comment on what the best instructions are for setting
> up xvfb and icewm?

If you do go down this route, the instructions you pasted above are
roughly the same commands.
I think it'd be better to use Xephyr though, because that will allow
you to see the test output (while still keeping it in a controlled
environment).

See the discussion under "Running tests" on the debugging page:
http://code.google.com/p/chromium/wiki/LinuxDebugging#Running_Tests
It links to more info about Xephyr, including the steps to set it up.

(...now that we use DumpRenderTree, I guess we should move the Xephyr
notes off of the layout tests page as it's no longer relevant to
layout tests in particualr.)

I would have hoped that we split the differences between running the tests

i)  in case there are regressions

ii) to figure out what the regressions actually mean

iii) to develop tests and check they are testing the right things

iv) to predict tree breakage in advance

for i) & iv) , Xvfb which produces no output should be the easiest/fairest.

for ii) & iii) Xephyr, Xtruss, Xgl, etc seem like they need referencing.

In general I favor things which reduce the noise in the signal ... both less extraneous/irrelevant failures and messages/output.

FWIW,

Peter.