CQ stats for the week of September 19

7 views
Skip to first unread message

Paweł Hajdan, Jr.

unread,
Sep 26, 2016, 8:02:05 PM9/26/16
to blink-dev
Statistics for project chromium
only for paths in the following set:
  third_party/WebKit
from 2016-09-19 00:00:00 till 2016-09-26 00:00:00 (UTC time).

CQ users:         128 out of    129 total committers  99.22%
  Committed       305 out of    312 commits           97.76%. 

 259 issues (289 patches) were tried by CQ, resulting in 347 attempts.
 242 patches (83.7% of tried patches, 69.7% of attempts) were committed by CQ,

False Rejections:
  37 attempts (12.6% of 293 attempts in 27 committed patchsets) were false rejections
   2 attempts (0.7% of 293 attempts in 2 committed patchsets) were infra false rejections

Patches which eventually land percentiles:
 10:  0.1 hrs,  1 attempts,  0 tryjob retries,  0 global retry quota
 25:  0.2 hrs,  1 attempts,  0 tryjob retries,  0 global retry quota
 50:  1.3 hrs,  1 attempts,  0 tryjob retries,  0 global retry quota
 75:  1.8 hrs,  1 attempts,  1 tryjob retries,  1 global retry quota
 90:  3.0 hrs,  2 attempts,  2 tryjob retries,  2 global retry quota
 95:  3.8 hrs,  2 attempts,  2 tryjob retries,  3 global retry quota
 99:  4.7 hrs,  3 attempts,  5 tryjob retries,  6 global retry quota
max:  7.3 hrs,  6 attempts,  8 tryjob retries, 12 global retry quota

Per-day stats:
  2016-09-19:   27 attempts; 50%  1.3; 90%  2.9; false rejections 16.0% ( 0.0% infra)
  2016-09-20:   71 attempts; 50%  0.8; 90%  2.2; false rejections  5.9% ( 0.0% infra)
  2016-09-21:   75 attempts; 50%  1.0; 90%  2.2; false rejections  2.9% ( 0.0% infra)
  2016-09-22:   66 attempts; 50%  0.9; 90%  2.5; false rejections  5.5% ( 0.0% infra)
  2016-09-23:   77 attempts; 50%  1.6; 90%  3.7; false rejections 11.1% ( 1.9% infra)
  2016-09-24:   23 attempts; 50%  1.1; 90%  3.1; false rejections 30.0% ( 0.0% infra)
  2016-09-25:    7 attempts; 50%  0.1; 90%  0.1; false rejections 20.0% ( 0.0% infra)

Top flaky builders (which fail and succeed in the same patch):
Master                         Builder                                                 Flakes            |Infra  |Compile|Test   |Invalid|Patch  |Other  
blink                          mac10.11_retina_blink_rel                                   1/    2 ( 50%)|   100%|     0%|     0%|     0%|     0%|     0%
master.chromium.win            win_chromium_rel_ng                                        93/  426 ( 22%)|    35%|     0%|    53%|     8%|     2%|     2%
master.chromium.android        linux_android_rel_ng                                       62/  384 ( 16%)|     3%|     0%|    73%|    21%|     3%|     0%
master.chromium.win            win_chromium_x64_rel_ng                                    55/  377 ( 15%)|    69%|     0%|     7%|    13%|     7%|     4%
chromium.win                   win_chromium_x64_rel_ng                                     1/    7 ( 14%)|     0%|   100%|     0%|     0%|     0%|     0%
master.chromium.mac            mac_chromium_10.10_rel_ng                                  26/  205 ( 13%)|   100%|     0%|     0%|     0%|     0%|     0%
master.chromium.android        android_n5x_swarming_rel                                   36/  356 ( 10%)|     0%|     0%|    53%|    39%|     6%|     3%
master.chromium.mac            mac_chromium_rel_ng                                        33/  359 (  9%)|    18%|     0%|    39%|    36%|     6%|     0%
master.chromium.win            win_clang                                                  29/  348 (  8%)|    72%|    10%|     0%|     0%|    14%|     3%
master.chromium.linux          linux_chromium_rel_ng                                      24/  349 (  7%)|     0%|     0%|    54%|    38%|     8%|     0%
master.chromium.linux          chromium_presubmit                                         32/  517 (  6%)|     0%|     0%|    81%|     0%|    19%|     0%
master.chromium.win            win_chromium_compile_dbg_ng                                20/  341 (  6%)|    75%|     0%|     0%|     0%|    10%|    15%
OVERALL                        OVERALL                                                   495/ 9972 (  5%)|    32%|     3%|    36%|    13%|    14%|     3%
master.chromium.android        android_clang_dbg_recipe                                    7/  328 (  2%)|    14%|    57%|     0%|     0%|    29%|     0%
master.chromium.android        android_arm64_dbg_recipe                                    7/  328 (  2%)|    14%|    57%|     0%|     0%|    29%|     0%
master.chromium.linux          linux_chromium_chromeos_rel_ng                              6/  329 (  2%)|     0%|     0%|    67%|     0%|    33%|     0%
master.chromium.linux          linux_chromium_chromeos_ozone_rel_ng                        6/  329 (  2%)|    17%|     0%|     0%|    50%|    33%|     0%
master.chromium.mac            ios-simulator                                               6/  329 (  2%)|     0%|     0%|     0%|     0%|    67%|    33%
master.chromium.linux          cast_shell_linux                                            6/  329 (  2%)|    50%|     0%|    17%|     0%|    33%|     0%
master.chromium.mac            ios-device                                                  5/  328 (  2%)|     0%|     0%|     0%|     0%|    80%|    20%
master.chromium.android        android_cronet                                              4/  323 (  1%)|    25%|    25%|     0%|     0%|    50%|     0%
master.chromium.linux          chromeos_amd64-generic_chromium_compile_only_ng             4/  325 (  1%)|    50%|     0%|     0%|     0%|    50%|     0%
master.chromium.linux          linux_chromium_compile_dbg_ng                               4/  325 (  1%)|    25%|    25%|     0%|     0%|    50%|     0%
master.chromium.android        android_compile_dbg                                         4/  325 (  1%)|    50%|     0%|     0%|     0%|    50%|     0%
master.chromium.mac            mac_chromium_compile_dbg_ng                                 4/  328 (  1%)|     0%|    50%|     0%|     0%|    50%|     0%
master.chromium.linux          linux_chromium_asan_rel_ng                                  4/  329 (  1%)|     0%|     0%|    50%|     0%|    50%|     0%
master.chromium.mac            ios-simulator-swarming                                      2/  168 (  1%)|     0%|     0%|     0%|     0%|    50%|    50%
master.chromium.linux          chromeos_x86-generic_chromium_compile_only_ng               3/  324 (  1%)|    33%|     0%|     0%|     0%|    67%|     0%
master.chromium.linux          chromeos_daisy_chromium_compile_only_ng                     3/  324 (  1%)|    33%|     0%|     0%|     0%|    67%|     0%
master.chromium.linux          linux_chromium_chromeos_compile_dbg_ng                      3/  324 (  1%)|    33%|     0%|     0%|     0%|    67%|     0%
master.chromium.linux          linux_chromium_clobber_rel_ng                               2/  323 (  1%)|     0%|     0%|     0%|     0%|   100%|     0%
master.chromium.android        cast_shell_android                                          2/  323 (  1%)|     0%|     0%|     0%|     0%|   100%|     0%
master.chromium.linux          blimp_linux_dbg                                             1/  167 (  1%)|     0%|     0%|     0%|     0%|   100%|     0%

Slowest CLs:

Per-day slowest CLs:
  2016-09-19:
  2016-09-20:
  2016-09-21:
  2016-09-22:
  2016-09-23:
  2016-09-24:
  2016-09-25:

Paweł

Philip Rogers

unread,
Sep 26, 2016, 8:34:12 PM9/26/16
to Paweł Hajdan, Jr., blink-dev, Chris Harrelson
Hi Paweł,

I love that the infra team tracks this data in detail now, and these updates are a useful summary. I rarely have the time to dig into the numbers in detail though... would it be possible to auto-generate 1-2 graphs and include them as a high-level summary at the top of the emails?

For example, a graph showing the weekly average cq latency and average commit count over the past ~6 weeks would be a useful high-level overview of our project's health.

Sergiy Byelozyorov

unread,
Sep 27, 2016, 12:01:12 PM9/27/16
to Philip Rogers, Paweł Hajdan, Jr., blink-dev, Chris Harrelson

Paweł Hajdan, Jr.

unread,
Oct 3, 2016, 11:36:27 AM10/3/16
to Philip Rogers, chrome-infrastructure-team, Chris Harrelson
[bcc:blink-dev, +chrome-infrastructure-team]

On Tue, Sep 27, 2016 at 2:33 AM, Philip Rogers <p...@chromium.org> wrote:
I love that the infra team tracks this data in detail now, and these updates are a useful summary. I rarely have the time to dig into the numbers in detail though... would it be possible to auto-generate 1-2 graphs and include them as a high-level summary at the top of the emails?

Thanks! Graphs and automating things more would sound great to me. What's the best way which we could share publicly?
 
For example, a graph showing the weekly average cq latency and average commit count over the past ~6 weeks would be a useful high-level overview of our project's health.

Agreed. FWIW I've been tracking historical values (for chromium CQ) manually in https://docs.google.com/a/google.com/spreadsheets/d/1l4baZasQA4ZkaNL8cRuHXowMxSt9l8GoDFml8XZIW1c/edit?usp=sharing (Google-internal).

Started https://goto.google.com/nadrn (Google-internal) for commit count.

For auto-graphing weekly cq latency, it seems we'd need to overhaul our monitoring for this. Current graphs I'm aware of don't match data reported by cq_stats.

Paweł
Reply all
Reply to author
Forward
0 new messages