Cannot understand flakiness dashboard

[Mikel Astiz]

Hi all,

I'm wondering who maintains or knows the flakiness dashboard in detail.

I'm investigating certain reported flakes (example), where tryserver.chromium.android is marked as VERYFLAKY, and hovering over a black cell says something like "PASS PASS PASS .", and I fail to find an example of a test failure.

Build logs (example) show failures (and flakes) for steps "with patch", but that doesn't seem a strong indicator of flakiness on trunk (without patch), is it?

The test (org.chromium.chrome.browser.sync.SyncCustomizationFragmentTest#testSetupOnly) does however get listed as failure in "retry summary" (with and without patch). Where does this come from? Is this WAI?

Thanks!

Mikel

[Yuke Liao]

I'm not sure who maintains flakiness dashboard either, but feel free to file a bug using component: Infra>Flakiness>Dashboard.

+Erik Chen, I think this might be related to your 'retry with patch' change. 

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAAy826erMjQnhBHZ0wY3QFqwNFihAVRSDrY7afWAHvi7-mNVow%40mail.gmail.com.

[Erik Chen]

I also don't know how to interpret this dashboard. :(

In the example you linked, there are a lot tests with "TEST RESULTS WERE INVALID". That might also be causing problems?

[Dirk Pranke]

The app is owned by Sean McCullough and his team in Ops, but it's been very badly understaffed for a long time, and doesn't work that well in some scenarios. 

I know how it works and can answer some questions, as can a few other people. 

Other comments inline ...

On Mon, Oct 29, 2018 at 8:34 AM, Erik Chen <erik...@chromium.org> wrote:

I also don't know how to interpret this dashboard. :(

In the example you linked, there are a lot tests with "TEST RESULTS WERE INVALID". That might also be causing problems?

On Mon, Oct 29, 2018 at 11:01 AM, Yuke Liao <liao...@chromium.org> wrote:

I'm not sure who maintains flakiness dashboard either, but feel free to file a bug using component: Infra>Flakiness>Dashboard.

+Erik Chen, I think this might be related to your 'retry with patch' change. 

On Mon, Oct 29, 2018 at 6:45 AM Mikel Astiz <mas...@chromium.org> wrote:

Hi all,

I'm wondering who maintains or knows the flakiness dashboard in detail.

I'm investigating certain reported flakes (example), where tryserver.chromium.android is marked as VERYFLAKY, and hovering over a black cell says something like "PASS PASS PASS .", and I fail to find an example of a test failure.

The second most recent black/VERYFLAKY cell I see is from:

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/android-kitkat-arm-rel/113969

Looking at the log for that test, I can't make heads nor tails of it, since it looks like it's running the test somewhere between 4 and 12 times even though it passes every time. So, someone who is more familiar with Android testing should probably take a look.

 

Build logs (example) show failures (and flakes) for steps "with patch", but that doesn't seem a strong indicator of flakiness on trunk (without patch), is it?

Most of the time, test flakiness on a try job is unrelated to the patch in question, and so flakiness on a "with patch" run can be reflective of flakiness on trunk, though it's not a guarantee.

 

The test (org.chromium.chrome.browser.sync.SyncCustomizationFragmentTest#testSetupOnly) does however get listed as failure in "retry summary" (with and without patch). Where does this come from? Is this WAI?

I'm not sure, as per my comment above. Something seems wrong somewhere :). But I'm not sure which things.

Can you file a bug and we can get someone to look into it in more detail? Or, link us to an existing bug if that's what triggered your investigation?

-- Dirk

 

Thanks!

Mikel

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAAy826erMjQnhBHZ0wY3QFqwNFihAVRSDrY7afWAHvi7-mNVow%40mail.gmail.com.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEYHnr2Kta9pOZfcvdXE7kWc_mVqwL8JYwtMLcayvVk0q8wuqQ%40mail.gmail.com.

[Mikel Astiz]

Hi,

Thanks for your replies, answers inline.

On Tue, Oct 30, 2018 at 11:20 PM Dirk Pranke <dpr...@chromium.org> wrote:

The app is owned by Sean McCullough and his team in Ops, but it's been very badly understaffed for a long time, and doesn't work that well in some scenarios. 

I know how it works and can answer some questions, as can a few other people. 

Other comments inline ...

On Mon, Oct 29, 2018 at 8:34 AM, Erik Chen <erik...@chromium.org> wrote:

I also don't know how to interpret this dashboard. :(

In the example you linked, there are a lot tests with "TEST RESULTS WERE INVALID". That might also be causing problems?

On Mon, Oct 29, 2018 at 11:01 AM, Yuke Liao <liao...@chromium.org> wrote:

I'm not sure who maintains flakiness dashboard either, but feel free to file a bug using component: Infra>Flakiness>Dashboard.

+Erik Chen, I think this might be related to your 'retry with patch' change. 

On Mon, Oct 29, 2018 at 6:45 AM Mikel Astiz <mas...@chromium.org> wrote:

Hi all,

I'm wondering who maintains or knows the flakiness dashboard in detail.

I'm investigating certain reported flakes (example), where tryserver.chromium.android is marked as VERYFLAKY, and hovering over a black cell says something like "PASS PASS PASS .", and I fail to find an example of a test failure.

The second most recent black/VERYFLAKY cell I see is from:

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/android-kitkat-arm-rel/113969

Looking at the log for that test, I can't make heads nor tails of it, since it looks like it's running the test somewhere between 4 and 12 times even though it passes every time. So, someone who is more familiar with Android testing should probably take a look.

 

Build logs (example) show failures (and flakes) for steps "with patch", but that doesn't seem a strong indicator of flakiness on trunk (without patch), is it?

Most of the time, test flakiness on a try job is unrelated to the patch in question, and so flakiness on a "with patch" run can be reflective of flakiness on trunk, though it's not a guarantee.

That makes sense. My question was whether the flakiness dashboard and related tools would report it as a flake.

 

 

The test (org.chromium.chrome.browser.sync.SyncCustomizationFragmentTest#testSetupOnly) does however get listed as failure in "retry summary" (with and without patch). Where does this come from? Is this WAI?

I'm not sure, as per my comment above. Something seems wrong somewhere :). But I'm not sure which things.

Can you file a bug and we can get someone to look into it in more detail? Or, link us to an existing bug if that's what triggered your investigation?

I filed a new bug as a blocker for our original bug.

Thanks,

Mikel

[Dirk Pranke]

On Tue, Oct 30, 2018 at 11:51 PM, Mikel Astiz <mas...@chromium.org> wrote:

Hi,

Thanks for your replies, answers inline.

On Tue, Oct 30, 2018 at 11:20 PM Dirk Pranke <dpr...@chromium.org> wrote:

The app is owned by Sean McCullough and his team in Ops, but it's been very badly understaffed for a long time, and doesn't work that well in some scenarios. 

I know how it works and can answer some questions, as can a few other people. 

Other comments inline ...

On Mon, Oct 29, 2018 at 8:34 AM, Erik Chen <erik...@chromium.org> wrote:

I also don't know how to interpret this dashboard. :(

In the example you linked, there are a lot tests with "TEST RESULTS WERE INVALID". That might also be causing problems?

On Mon, Oct 29, 2018 at 11:01 AM, Yuke Liao <liao...@chromium.org> wrote:

I'm not sure who maintains flakiness dashboard either, but feel free to file a bug using component: Infra>Flakiness>Dashboard.

+Erik Chen, I think this might be related to your 'retry with patch' change. 

On Mon, Oct 29, 2018 at 6:45 AM Mikel Astiz <mas...@chromium.org> wrote:

Hi all,

I'm wondering who maintains or knows the flakiness dashboard in detail.

I'm investigating certain reported flakes (example), where tryserver.chromium.android is marked as VERYFLAKY, and hovering over a black cell says something like "PASS PASS PASS .", and I fail to find an example of a test failure.

The second most recent black/VERYFLAKY cell I see is from:

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/android-kitkat-arm-rel/113969

Looking at the log for that test, I can't make heads nor tails of it, since it looks like it's running the test somewhere between 4 and 12 times even though it passes every time. So, someone who is more familiar with Android testing should probably take a look.

 

Build logs (example) show failures (and flakes) for steps "with patch", but that doesn't seem a strong indicator of flakiness on trunk (without patch), is it?

Most of the time, test flakiness on a try job is unrelated to the patch in question, and so flakiness on a "with patch" run can be reflective of flakiness on trunk, though it's not a guarantee.

That makes sense. My question was whether the flakiness dashboard and related tools would report it as a flake.

Good question. I'm not sure of the answer but maybe liaoyuke@ or stgao@ knows. 

-- Dirk

[Yuke Liao]

Hi Mikel,

FindIt's flake detector did detect this flake, please see:

https://findit-for-me.appspot.com/flake/occurrences?key=ag9zfmZpbmRpdC1mb3ItbWVyewsSBUZsYWtlInBjaHJvbWl1bUBjaHJvbWVfc3luY19zaGVsbF90ZXN0X2Fwa0BvcmcuY2hyb21pdW0uY2hyb21lLmJyb3dzZXIuc3luYy5TeW5jQ3VzdG9taXphdGlvbkZyYWdtZW50VGVzdCN0ZXN0U2V0dXBPbmx5DA

Unfortunately, there is no easy way to navigate to this link as of now, the https://findit-for-me.appspot.com/ranked-flakes dashboard only surfaces flakes that meet certain criteria, and it seems this test doesn't. We're still improving the tool and will fix that, at least make tests that don't meet the criteria searchable, please let us know if you have any other feedback.

[Shuotao Gao]

For this particular build (build time on 10/26/2018) mentioned in this email thread, Flake Detector failed to detect it unfortunately.

Flake Detector was broken for an extended period of time by "retry with patch", we have fixed the majority of cases there, but this is another edge case that is not handled yet.

I filed https://crbug.com/901158 to follow up.