PSA: mac_chromium_rel_ng lagging behind due to persistent browser_tests failures

[John Budorick]

Hi folks,

As some of you may have noticed, mac_chromium_rel_ng is having a bad day (as are the Mac CI bots) thanks to some issues with browser_tests. We're working on identifying and either resolving or working around the root cause. Please bear with us. I'll update this thread either once things are fixed or at 5pm PDT, whichever happens earlier.

Thanks,

John

[John Budorick]

mac_chromium_rel_ng is still behind, and we haven't yet been able to identify or address the underlying issue. Dropping browser_tests on the bot to experimental while continuing to investigate.

[Nico Weber]

Is there a bug for this?

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAOee4m5%3DWxXRzVftG00mh3o3i6JB6ubj28OxS-heyB-1nCK5uQ%40mail.gmail.com.

[John Budorick]

At this point, there are several covering slightly different symptoms: crbug.com/767397, crbug.com/825215, crbug.com/828031

[Dirk Pranke]

Whatever is going on seems to now aggressively be taking out the Mac fleet, so I'm adopting a much more aggressive treatment.

As of now, I've removed all variants of browser_tests from all Mac bots (both CQ and CI) across all versions, to see if that'll keep things from dying.

This is obviously not a good state to be in, and we are working to split the swarming pools up so that we can run some tests in dedicated pools to get coverage back without killing the rest of the fleet, and that should give us a mechanism to segregate potentially problematic traffic from believed-good traffic while continuing to hunt further.

You can star crbug.com/828031 for updates, which I've escalated to P0 and own for now.

-- Dirk

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAOee4m6uvTwfctAZ73WFKrmk%3Dinui6D4GRWkK_BL-iaywYMMGw%40mail.gmail.com.

[Dirk Pranke]

Update: we've split up the pools and are testing reenabling just the browser_tests (not viz_browser_tests or any other variant) on just chromium.mac and the matching trybots. If that works, we'll also reenable the other test suites on the main bots (but not the fyi/memory/clang bots yet), keep an eye on things overnight and work on adding more test suites tomorrow and using this mechanism to help troubleshoot what's going on. 

We don't want to re-enable everything just yet because we don't know if we got the pool sizes right.

-- Dirk

[Dirk Pranke]

Update: there are still some capacity constraints and I've disabled the browser_tests on 10.10. It's likely there will be issues on the 10.10 and 10.11 waterfall bots until tomorrow, but I think the CQ and 10.12/10.13 should be okay.

-- Dirk

[Nico Weber]

viz_browser_tests got enabled 3 days ago. That's after this thread started, but if things got a lot worse recently it might be due to that (https://chromium-review.googlesource.com/c/chromium/src/+/1095687).

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTAk1XREmWmUZYBG_2d_oErvEYBq1nbZ%3DYzuMLUnO4oC6A%40mail.gmail.com.