Fwd: Need volunteer to help with interactive_ui_tests on Windows

[Jói Sigurðsson]

Hi all,

The interactive_ui_tests are badly broken, and have been for 24 hours
or so, with no obvious culprits in the blamelists from when they
started flaking, and with tricks like restarting the bot not working.

If anybody who knows anything about the test server used by the
interactive UI tests feels like being a hero and helping the sheriffs
out, please take a look at http://crbug.com/106543.

Cheers,
Jói

[Marc-Antoine Ruel]

Often for issues like that, it's a good idea to try to reproduce locally.

Failing that, you can try to bisect revisions with try jobs

 -b win_rel,win -t interactive_ui_tests -r 123

M-A

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
   http://groups.google.com/a/chromium.org/group/chromium-dev

[Jenn Braithwaite (胡慧鋒)]

Don't know if this is related, but I spotted some failures yesterday, thinking it was specific to panels and reported in http://crbug.com/106489

Upon investigation, dimich and I saw that many interactive_ui_tests (non-panels) were also failing. 

cc-ing sky who made the test startup changes.

Jenn

[Paweł Hajdan, Jr.]

How about inserting more logging to the test server code (both Python and C++ one)?

I can happily review changes (OWNER there). Please avoid any weird TBRs to the test server code. If you need something done more quickly, remember that net/ OWNERS can also review the test server code.

On Tue, Dec 6, 2011 at 16:41, Jói Sigurðsson <j...@google.com> wrote:

Hi all,

The interactive_ui_tests are badly broken, and have been for 24 hours
or so, with no obvious culprits in the blamelists from when they
started flaking, and with tricks like restarting the bot not working.
If anybody who knows anything about the test server used by the
interactive UI tests feels like being a hero and helping the sheriffs
out, please take a look at http://crbug.com/106543

Cheers,
Jói

[Scott Violet]

I added comments to the bug. Failures like that generally indicate a
test is causing focus to go to another app so that the next test to
run doesn't get focus.

-Scott

[Jói Sigurðsson]

I am still looking for a volunteer to help with this bug. Windows and
interactive_ui_tests experience is desired.

I just got around to focusing on the issue myself about an hour ago
after getting a few other problems on the tree under control, and it's
getting near the end of my shift. So far I've been trying to
reproduce locally, in the hope that I could then bisect, with no luck.

Is anybody in the situation of being able to bisect on the bot itself?
Alternatively, if anybody has been able to reproduce locally, please
bisect.

Cheers,
Jói

[Scott Violet]

I wonder if it's really taking > 10 seconds for the server to startup
on this machine. It might be worth upping the timeout and seeing if
that causes more tests to pass.

-Scott

[Scott Graham]

I used http://www.jam-software.com/heavyload/HeavyLoadSetup.exe to get
the failures to reproduce (sometimes) locally. Probably just building
another tree at the same time as running the tests would work too.

So, increasing the timeout seems like a plausible workaround. Maybe
there's some reason why load on those machines increased around that
time?

scott

----------------------------

e.g.

[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from BrowserFocusTest, where TypeParam =
[ RUN ] BrowserFocusTest.FocusTraversal
[10468:10380:1208/190240:606624317:WARNING:test_server_win.cc(35)] Timeout reach
ed; unblocking pipe by writing 4 bytes
[10468:9436:1208/190240:606624317:ERROR:test_server_win.cc(76)] Timeout exceeded
for ReadData
[10468:9436:1208/190240:606624317:ERROR:test_server_win.cc(159)] Could not read
server_data_len
.\browser\browser_focus_uitest.cc(454): error: Value of: test_server()->Start()
Actual: false
Expected: true
[ FAILED ] BrowserFocusTest.FocusTraversal, where TypeParam = and GetParam()
= (12141 ms)
...

[Scott Violet]

sergeyu uped the timeout and now it seems only some a11y tests are
timing out. I'm going to disable them and hopefully we'll see green
soon.

-Scott

[Mark Larson (Google)]

Thanks to Sergey and Scott for diving into this (and Joi, too). I don't think we're out of the woods yet, but I'm, not ready to cry (cry for help, of course) like I was yesterday.

I'd like to prevent this many days of a broken test suite in the future. Is there something we can do to get interactive_ui_tests off the waterfall (moving the tests to some other suite... not interested in losing coverage or tests)? Maybe that's not the right question, but it sure feels like these tests are mysterious, easily flakable, and hard to keep green. Maybe the right question is "how can we make it easier to write non-flaky interactive UI tests?"

--Mark

[Paweł Hajdan, Jr.]

On Fri, Dec 9, 2011 at 07:09, Mark Larson (Google) <m...@chromium.org> wrote:

Is there something we can do to get interactive_ui_tests off the waterfall (moving the tests to some other suite... not interested in losing coverage or tests)?

If we can do that, we should. Generally being in interactive_ui_tests increases chance of being flaky.

 

Maybe that's not the right question, but it sure feels like these tests are mysterious, easily flakable, and hard to keep green. Maybe the right question is "how can we make it easier to write non-flaky interactive UI tests?"

Note that in this case the cause was a Python test server, which failed to start in time. It's not specific to interactive_ui_tests, and could happen with any other test suite.

In fact, the applied change (http://codereview.chromium.org/8872050) re-introduces another problem (http://code.google.com/p/chromium/issues/detail?id=93770) for which the fix was http://codereview.chromium.org/7744004

I wonder why my advice to add more debugging (logging) code has been ignored while we still had a good repro case.

Anyway, I don't think the change applied (http://codereview.chromium.org/8872050) is correct, and instead suggest finding the root cause. Increasing action_timeout_ms isn't an optimal option either, because it's going to make other tests slower. Furthermore the timeout is inconsistent between POSIX and Windows test server.

Bottom line: root cause is still not identified, and problem is still there. I suggest to focus on this test server problem for now instead of more vaguely defined interactive_ui_tests problem, which of course is still there.

[Scott Violet]

As Pawel mentions, this time around the failure has not been because
of the tests themselves, but rather problems in starting the test
server that some of the tests rely on. I don't know what's unique
about this bot and why the problem was so persistent here.
Marc-Antoine said yesterday this machine shouldn't be that different
from the other machines we have and that it didn't look like the load
was high (it has 2 dedicated cores).

We had a similar problem last month. Similar in that interactive ui
tests were continually failing on the try bots for a couple of days
and sporadically on the watefall (see
http://code.google.com/p/chromium/issues/detail?id=102297 ). That
turned out to be a bug in one test that effected the system in such a
way that other tests would fail. That possibility still exists (I
disabled the failing test at the time). So, there is definitely room
for improvement in the interactive ui test suites.

I'm tempted to say we need owners for test suites, maybe with some
sort of rotation... But I know how well that would go over.

-Scott

[Jenn Braithwaite (胡慧鋒)]

On Fri, Dec 9, 2011 at 3:17 AM, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

On Fri, Dec 9, 2011 at 07:09, Mark Larson (Google) <m...@chromium.org> wrote:

Is there something we can do to get interactive_ui_tests off the waterfall (moving the tests to some other suite... not interested in losing coverage or tests)?

If we can do that, we should. Generally being in interactive_ui_tests increases chance of being flaky.

 We have a bunch of panel browser tests that were moved into interactive_ui_tests simply to avoid being run in parallel with other tests (messes with focus/active status). If anyone knows of another option for browser_tests that have this requirement, we would be able to move our tests out of interactive_ui_tests.

Thanks,

Jenn

[Scott Violet]

interactive ui tests is intended for tests like this.

-Scott

[Paweł Hajdan, Jr.]

Yes, and to make it clear - it's being focus/timing dependent that makes a test flaky, not the name of the test binary.