CQ does not match the list of "reverting" bots

[Dmitry Gozman]

Hi blink-infra,

Recently I had a couple of reverts due to some random bot failing, which is not available on CQ and trybots. Here are two examples of my patches, but I've also seen the same bots affecting other developers.

Sheriffs guideline for any red bot on sherff-o-matic is "in all cases, the preferred way of dealing with alerts is to revert the offending patch via Gerrit, even if the failure does not close the tree."

I personally think this is not a sustainable practice, especially considering that there is no way to trigger all the rare bots with my patch (or I am not aware of how to do it). It is frustrating to blindly fight obscure compile failures, to say the least. Should we refine our guidelines and/or try to match CQ with waterfall bots?

Thanks,

Dmitry

[Robert Ma]

+chromium-dev (blink-infra is a rather inactive list; we don't really have a "Blink Infra" team and the scope of Blink Infra is more about Blink-specific tooling like layout tests.)

The two examples you gave are failures in blink_common_unittests and Jumbo build, respectively. And you're really asking a more generic question:

Shall we change the sheriff guidelines, or try to mirror the non-tree-closing bots on sherrif-o-matic to CQ as well?

Passing the question to more knowledge fellow Chromium developers :)

--
You received this message because you are subscribed to the Google Groups "blink-infra" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-infra...@chromium.org.
To post to this group, send email to blink...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-infra/CAJfDkaF%3DqCkm287rVcYHGDwjqZr3O3PUaG31kfKP3GbB22yUtA%40mail.gmail.com.

[Dirk Pranke]

We are supposed to have matching trybots for every builder that shows up in sheriff-o-matic. By and large, that is true (there may still be a couple of exceptions. If you find one, let us know and we will try to fix it ASAP).

Finding the matching trybot for a given waterfall builder is, however, much harder than it should be. The way to do it is to look up the entry for a given waterfall

builder in 

https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/trybots.py

The way we want to make it easier to find things is to simply have the waterfall builder and trybuilder name match.

For your first failure, where linux-jumbo-rel broke, that's already the case, because that's a newly added configuration. In addition, we also want linux-jumbo-rel to be in the CQ (because there are a lot of jumbo-specific breakages), but that's not quite configured yet. 

For your second failure, I would've expected that failure to have been caught by win_chromium_compile_dbg_ng in the CQ (which should effectively be the matching trybot for WebKit Win7 (dbg), and I'm not sure why it wasn't. Mind filing a bug for that?

We will likely always have some builders on the main waterfalls that aren't in the CQ. Some configurations are too slow to test in a short enough amount of time to be in the CQ, or they don't catch enough errors to make economic sense to run on every tryjob. Finding the right balance of CQ coverage to failures that get through is something we try to stay on top of but don't always succeed.

-- Dirk

On Fri, May 11, 2018 at 7:07 PM, Robert Ma <robe...@chromium.org> wrote:

+chromium-dev (blink-infra is a rather inactive list; we don't really have a "Blink Infra" team and the scope of Blink Infra is more about Blink-specific tooling like layout tests.)

The two examples you gave are failures in blink_common_unittests and Jumbo build, respectively. And you're really asking a more generic question:

Shall we change the sheriff guidelines, or try to mirror the non-tree-closing bots on sherrif-o-matic to CQ as well?

Passing the question to more knowledge fellow Chromium developers :)

On Fri, May 11, 2018 at 9:35 PM Dmitry Gozman <dgo...@chromium.org> wrote:

Hi blink-infra,

Recently I had a couple of reverts due to some random bot failing, which is not available on CQ and trybots. Here are two examples of my patches, but I've also seen the same bots affecting other developers.

Sheriffs guideline for any red bot on sherff-o-matic is "in all cases, the preferred way of dealing with alerts is to revert the offending patch via Gerrit, even if the failure does not close the tree."

I personally think this is not a sustainable practice, especially considering that there is no way to trigger all the rare bots with my patch (or I am not aware of how to do it). It is frustrating to blindly fight obscure compile failures, to say the least. Should we refine our guidelines and/or try to match CQ with waterfall bots?

Thanks,

Dmitry

--
You received this message because you are subscribed to the Google Groups "blink-infra" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-infra+unsubscribe@chromium.org.

To post to this group, send email to blink...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-infra/CAJfDkaF%3DqCkm287rVcYHGDwjqZr3O3PUaG31kfKP3GbB22yUtA%40mail.gmail.com.

--

You received this message because you are subscribed to the Google Groups "blink-infra" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-infra+unsubscribe@chromium.org.

To post to this group, send email to blink...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-infra/CAOPAaN%2Bvn5En%3DvtcVWEzHKXHi3upfLxmu7z6ySy-bRS_PucFSA%40mail.gmail.com.

[Marijn Kruisselbrink]

Re the blink_common_unittests case I also recently ran into the fact that apparently the target isn't build on most CQ bots (I think only on linux it is?), and isn't run even on that one... Probably just an unintentional oversight though.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTBGKGcRRQhZ%2BBtnFG9jsQv7HDJwLBdk1CDM%3D0e-q%2BZNhQ%40mail.gmail.com.

[Dirk Pranke]

The chromium.webkit builders should only be running the layout tests; all of the gtest-based tests should be running on the non-.webkit builders. It may be that someone added something in the wrong place and that's the gap. If so, it's easily fixable.

-- Dirk

[Dmitry Gozman]

Thank you for the answers, the link is super-useful!

- Dmitry