Re: [blink-dev] CQ stats for the week of Sep 22, 2014

[Paweł Hajdan, Jr.]

[+hackability-cy]

On Tue, Sep 30, 2014 at 12:50 AM, Sergey Berezin <sergey...@chromium.org> wrote:

Top flaky builders (which fail and succeed in the same patchset):

Builder Name                             Succeeded       Flaky Failures  Flakiness (%)  

----------------------------------------------------------------------------------------

win_blink_rel                            519             193             27.1           

android_chromium_gn_compile_rel          569             25              4.2            

mac_blink_rel                            548             18              3.2            

I'd like to highlight how win_blink_rel is significantly more flaky than anything else (27% vs 4% for the next builder).

This is actually a historical trend:

September 15: 30% vs. 3% (https://groups.google.com/a/chromium.org/d/msg/blink-dev/ATrLvGek8p4/kGjvgclfeG4J)

September 8: 29% vs. 3% (https://groups.google.com/a/chromium.org/d/msg/blink-dev/d7eOogmN1tg/jN02Lgt9MUEJ) 

September 1: 17% vs. 5% (https://groups.google.com/a/chromium.org/d/msg/blink-dev/whHnR0WcWQs/fS6JKPhYLNcJ)

August 25: 22% vs. 5% (https://groups.google.com/a/chromium.org/d/msg/blink-dev/LyRqs0nXYVE/CT0UDoECESQJ)

August 18: 25% vs. 21% (https://groups.google.com/a/chromium.org/d/msg/blink-dev/T-FHSqHT5h8/wCxOcBZMuPYJ)

August 11: 25% vs. 11% (https://groups.google.com/a/chromium.org/d/msg/blink-dev/PCwySDhfqHU/JygICEwCGqMJ)

August 4: 44% vs. 39% (https://groups.google.com/a/chromium.org/d/msg/blink-dev/Iqr7fROW6Gw/Hnnvw-MlwoQJ)

July 28: 28% vs. 18% (https://groups.google.com/a/chromium.org/d/msg/blink-dev/wTnravJ7FJk/A_ZO9WZycZQJ)

July 21: 29% vs. 13% (https://groups.google.com/a/chromium.org/d/msg/blink-dev/gdxK1b7rsKo/KduVonNCjxYJ)

July 14: 27% vs. 12% (https://groups.google.com/a/chromium.org/d/msg/blink-dev/bYT-p3Z2qYM/b3PQrnctd9IJ)

Some other highlights:

1. win_blink_rel has been consistently top #1 flaky builder on blink

2. since approx. August 25 flakiness of other builders has been reduced to no more than 5% (which is great), but win_blink_rel stayed more or less the same

We had a discussion on infra-dev about this (https://groups.google.com/a/chromium.org/d/msg/infra-dev/lPw3p7hZNF0/p7Bxjg-lgm8J) and I think the main takeaway is that the Blink community needs to make layout tests on Windows more reliable. It's possible only a subset of developers works on Windows layout tests locally, but almost everyone uses CQ and so this impacts almost everyone - specifically, changes are getting falsely rejected because of these flakes, or take longer because of CQ retrying the flakes. Furthermore, we likely wouldn't be able to meet our CY goals for Blink CQ without this improving - there are limits of how fast and reliably we can run tests which are just not that reliable. Please help us.

Paweł

[John Abd-El-Malek]

+1 to all this, this is what I wrote earlier. However this message should probably be sent to blink-dev.

 

Furthermore, we likely wouldn't be able to meet our CY goals for Blink CQ

The CY goals were for the chromium CQ, not the blink one.

 

without this improving - there are limits of how fast and reliably we can run tests which are just not that reliable. Please help us.

Paweł

--
You received this message because you are subscribed to the Google Groups "Chromium Hackability Code Yellow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hackability-c...@chromium.org.
To post to this group, send email to hackabi...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/hackability-cy/CAATLsPYo0BiaPygPGRH4ZeRqzHtKW4OnAHLqusP5FPYnx1K6wQ%40mail.gmail.com.