Slow tests that timeouts

[Weizhong Xia]

Dear blink-devs,

As another effort trying to speed up blink_web_tests on CQ, We are looking at slow tests that also timeouts. From the timing stats(example) and some research, there are about 0.25% such tests, but running such tests takes about 4% of the total run time. So we are thinking of doing something about it.

We would like to propose as below, and get feedback on this:

1. We add Skip to test expectations for slow tests that also timeouts. This will effectively skip all such tests on swarming bots, and save ~50s real time on linux-rel.

2. We add a presubmit check to make any unskipped Slow+Timeout an error. (We can also make this a warning, but such warning will persist, so kind of annoying)

3. Devs are encouraged to take actions to either fix the timeout, or break the slow tests into pieces to make it not slow.

Please let us know your opinions on this.

Thanks, Weizhong

[Stephen Chenney]

I recently found a test that was TIMEOUT in TestExpectations, and still timed out in Slow, so I left it in TestExpectations under the assumption that it will timeout faster there.

The underlying issue is not a slow test but a test that hangs and we may or may not want to skip such a test. Would identifying these cases and moving them from Slow to TestExpectations have any noticeable impact?

 

Please let us know your opinions on this.

Thanks, Weizhong

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSipU3NDSp7tWd%3DqA%3DrZWMrmXjfT4NzTG7U8EoEAgsQ1QXw%40mail.gmail.com.

[Weizhong Xia]

>> I recently found a test that was TIMEOUT in TestExpectations, and still timed out in Slow, so I left it in TestExpectations under the assumption that it will timeout faster there.

What do you mean "timed out in Slow"? A test did not mark as slow, but timeout after 60 seconds? Worth mentioning some slow wpt tests are given the same timeout value as slow tests in the code.

But I did see some tests timed out at some abnormal value. For example tests below timeouts at 22 second. Not sure how this could happen.

19:13:50.493 20120 [3522/4178] external/wpt/webrtc-identity/idlharness.https.window.html failed unexpectedly (test timed out) 22.3300s

[Stephen Chenney]

On Wed, Aug 18, 2021 at 3:06 PM Weizhong Xia <weiz...@google.com> wrote:

>> I recently found a test that was TIMEOUT in TestExpectations, and still timed out in Slow, so I left it in TestExpectations under the assumption that it will timeout faster there.

What do you mean "timed out in Slow"? A test did not mark as slow, but timeout after 60 seconds? Worth mentioning some slow wpt tests are given the same timeout value as slow tests in the code.

See https://chromium-review.googlesource.com/c/chromium/src/+/3097709/1 and the blink_win7 try job.

I moved the expectation for external/wpt/css/css-fonts/test_font_family_parsing.html from the TestExpectations file to the SlowTests file and ran the try jobs. The test was still slower than the slow test timeout, and given the console output I strongly suspect the test hangs indefinitely.

[Xianzhu Wang]

I see the CL changed the expectation of some tests from [ Timeout ] to [ Slow ], and the tests still timeout. To make the bot green, we need to choose from:

a. Add back the [ Timeout ] entries in TestExpectations with the [ Slow ] entries in SlowTests;

b. Revert the change for the tests (i.e. keep the [ Timeout ] entries in TestExpectations without [ Slow ] entries in SlowTests);

c. Add [ Skip ] for the test. We can keep both [ Timeout ] and [ Slow ] as they don't matter if there is [ Skip ], but they partly describe the reason for the skip.

I think Weizhong's proposal is to handle #a which makes the tests unnecessarily take a long time before timeout. Proposal #1 changes #a to #c, and proposal #2 disallows #a.

For now it's not easy to check if a timing-out test is just slow or it hangs indefinitely. @Brian Sheedy is working on a project to automatically update TestExpectations based on bot history by removing stale failure/flaky entries for passing tests, to recover test coverage. I think the project could be extended in the future to handle slow/timeout entries based on bot history. For example, we can force running tests with Skip, and try bigger timeout for Timeout, on some bots periodically, to check 1) if a timing-out test is just slow, and 2) if a skipped test can be unskipped, and update test expectations automatically.