Significant flakiness on CQ

[Kenneth Russell]

Fellow Chromium developers, and sheriffs,

Significant flakiness appears to have crept in to a couple of the test suites recently. There are large numbers of test failures on a couple of the bots. These are significant enough that I just filed two P0s about them:

net_unittests flaky on android_n5x_swarming_rel

http://crbug.com/665686

webkit_tests failing >100 tests and aborting on linux_chromium_rel_ng

http://crbug.com/665693

I'd appreciate any help triaging these failures. They're preventing CLs from landing.

Thanks,

-Ken

P.S. Other bugs that affected my current CL:

http://crbug.com/665688

http://crbug.com/661434

It looks like we as a team need a stronger focus on eliminating flakiness.

[Xida Chen]

This might help:

https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Linux%20Trusty/builds/19980

Looks like the webkit_tests didn't start properly. The build log shows that it takes 39 seconds to finish the webkit_tests, and normally it should take more than 20 mins.

qyearsley@: could this be infra related?

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

[Kenneth Russell]

Thanks for the pointer Xida. That does help narrowing down the root cause.

There is no chance this is an infra problem. It's definitely caused by a code change in the Chromium tree.

On Tue, Nov 15, 2016 at 7:09 PM, Xida Chen <xida...@chromium.org> wrote:

This might help:

https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Linux%20Trusty/builds/19980

Looks like the webkit_tests didn't start properly. The build log shows that it takes 39 seconds to finish the webkit_tests, and normally it should take more than 20 mins.

qyearsley@: could this be infra related?

On Tue, Nov 15, 2016 at 9:57 PM Kenneth Russell <k...@chromium.org> wrote:

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

[Kenneth Russell]

FYI, I spot-checked several builds on https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng?numbuilds=200 and the only jobs making it through the CQ are those which don't trigger webkit_tests.

All webkit_tests jobs on this configuration are failing. This is a major outage. The tree should probably be closed in response.

I'm trying to build blink_tests locally on my Linux workstation but something seems to be wrong with goma and/or my machine's configuration and it's been going for over a half hour.