CQ stats for the week of May 18

11 views
Skip to first unread message

Paweł Hajdan, Jr.

unread,
May 26, 2015, 3:44:45 AM5/26/15
to chromium-dev
Statistics for project chromium
from 2015-05-17 23:03:09.826820 till 2015-05-24 12:59:24.131260 (local time).

CQ users:         329 out of    336 total committers  97.92%
  Committed       906 out of    934 commits           97.00%.

Bots:                     3 out of    336 total committers   0.89%
  Committed by CQ        69 out of    934 commits            7.39%
  Committed directly      0 out of    934 commits            0.00%

Manual committers:     16 out of all    336 users     4.76%
  Committed            28 out of        934 commits   3.00%

Total time tree open: 146.5 hours of 168.0 hours (87.18%).

1031 issues (1195 patches) were tried by CQ, resulting in 1161 attempts.
 798 patches (66.8% of tried patches, 68.7% of attempts) were committed by CQ,

Rejections:
 256 attempts (22.0% of 1161 attempts) were unsuccessful in 235 uncommitted patches
  This includes:
    76 attempts (6.5% of 1161 attempts) stopped manually (CQ box was unchecked) in 74 uncommitted patches
     8 attempts (0.7% of 1161 attempts) are missing LGTM in 7 uncommitted patches
     3 attempts (0.3% of 1161 attempts) failed to apply patch in 3 uncommitted patches
    34 attempts (2.9% of 1161 attempts) failed presubmit bot (often due to missing OWNERS LGTM) in 29 uncommitted patches
   135 attempts (11.6% of 1161 attempts) failed jobs (excluding presubmit): 223 in 129 uncommitted patches

False Rejections:
  22 attempts (1.9% of 1161 attempts) were false rejections in 20 committed patches
  False rejections include:
    22 attempts (1.9% of 1161 attempts) failed jobs (excluding presubmit): 22 in 20 committed patches
  Other rejections in committed patches for valid reasons:
    45 attempts (3.9% of 1161 attempts) stopped manually (CQ box was unchecked) in 44 committed patches
    10 attempts (0.9% of 1161 attempts) are missing LGTM in 9 committed patches
    30 attempts (2.6% of 1161 attempts) failed presubmit bot (often due to missing OWNERS LGTM) in 28 committed patches

Mean duration in CQ trying a patch:
  wallclock:        3:37:33 (218 min).
  sum of attempts:  1:22:57 ( 83 min).

Median duration in CQ trying a patch:   
  wallclock:        1:01:07 ( 61 min).  
  sum of attempts:  0:57:18 ( 57 min).  

Patches which eventually land percentiles:
10:  0.1 hrs,  1 attempts
25:  0.4 hrs,  1 attempts
50:  1.0 hrs,  1 attempts
75:  1.8 hrs,  1 attempts
90:  2.8 hrs,  2 attempts
95:  3.7 hrs,  2 attempts
99:  5.2 hrs,  3 attempts

Slowest CLs:

Top flaky builders (which fail and succeed in the same patch):
Builder Name                           Succeeded       Flaky Failures  Flakiness (%)
chromium_presubmit                      1696              26             1.51%
android_clang_dbg_recipe                1599              16             0.99%
mac_chromium_rel_ng                     1465               2             0.14%
linux_chromium_gn_rel                   1517               2             0.13%
mac_chromium_compile_dbg_ng             1556               1             0.06%
linux_perf_bisect                         14               0             0.00%
linux_arm_compile                          2               0             0.00%
win_chromium_x64_rel_ng                 1470               0             0.00%
linux_chromium_chromeos_rel_ng          1500               0             0.00%
linux_chromium_compile_dbg_32_ng        1604               0             0.00%
linux_chromium_gn_dbg                   1588               0             0.00%
android_arm64_dbg_recipe                1753               0             0.00%
linux_blink_rel                           59               0             0.00%
win8_chromium_gn_dbg                       2               0             0.00%
linux_chromium_chromeos_ozone_rel_ng    1535               0             0.00%
android_chromium_gn_compile_dbg         1669               0             0.00%
linux_android_rel_ng                    1519               0             0.00%
android_chromium_gn_compile_rel         1672               0             0.00%
win8_chromium_gn_rel                       4               0             0.00%
win_blink_rel                              2               0             0.00%
ios_rel_device_ninja                    1634               0             0.00%
android_compile_dbg                     1632               0             0.00%
cast_shell_linux                        1643               0             0.00%
ios_dbg_simulator_ninja                 1638               0             0.00%
android_nexus5_perf_bisect                 6               0             0.00%
linux_chromium_rel_ng                   1491               0             0.00%
linux_chromium_asan_rel_ng              1503               0             0.00%
cast_shell_android                      1708               0             0.00%
win8_chromium_rel                       1543               0             0.00%
mac_blink_rel                              2               0             0.00%
win_chromium_rel_ng                     1465               0             0.00%
linux_chromium_gn_chromeos_rel          1606               0             0.00%
mac_perf_bisect                           16               0             0.00%
linux_chromium_chromeos_compile_dbg_ng  1572               0             0.00%
linux_nacl_sdk_build                       2               0             0.00%
linux_chromium_clobber_rel_ng           1586               0             0.00%
win_perf_bisect                            4               0             0.00%
mac_chromium_gn_rel                        6               0             0.00%
linux_chromium_dbg_32_ng                   2               0             0.00%
win_chromium_compile_dbg_ng             1537               0             0.00%
mac_chromium_gn_dbg                        4               0             0.00%

Please file CQ bugs at crbug.com, labels "Infra" "Build-CommitQueue".

Paweł

Daniel Bratell

unread,
May 26, 2015, 5:20:11 AM5/26/15
to chromium-dev, Paweł Hajdan, Jr.
On Tue, 26 May 2015 09:43:35 +0200, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

Patches which eventually land percentiles:
10:  0.1 hrs,  1 attempts
25:  0.4 hrs,  1 attempts
50:  1.0 hrs,  1 attempts
75:  1.8 hrs,  1 attempts
90:  2.8 hrs,  2 attempts
95:  3.7 hrs,  2 attempts
99:  5.2 hrs,  3 attempts

While there is always room for improvement, I must comment that this is such a huge change to what it was like a year ago. 90% landed within 3 hours now, my feeling a year ago was that I was lucky if I could land something the same day.

The 0.5-2.5 hours that most patches need now, is that bot cycle time and queuing?

/Daniel

Paweł Hajdan, Jr.

unread,
Jun 1, 2015, 4:50:10 AM6/1/15
to Daniel Bratell, chromium-dev
On Tue, May 26, 2015 at 11:19 AM, Daniel Bratell <bra...@opera.com> wrote:
On Tue, 26 May 2015 09:43:35 +0200, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

Patches which eventually land percentiles:
10:  0.1 hrs,  1 attempts
25:  0.4 hrs,  1 attempts
50:  1.0 hrs,  1 attempts
75:  1.8 hrs,  1 attempts
90:  2.8 hrs,  2 attempts
95:  3.7 hrs,  2 attempts
99:  5.2 hrs,  3 attempts

While there is always room for improvement, I must comment that this is such a huge change to what it was like a year ago. 90% landed within 3 hours now, my feeling a year ago was that I was lucky if I could land something the same day.

Yay, thanks! Yeah it was much worse, with 50% starting at about 5 hours. Swarming helped reduce bot cycle time a lot, and there were also several improvements to CQ code and recipes.
 
The 0.5-2.5 hours that most patches need now, is that bot cycle time and queuing?

It's mostly bot cycle time and retries (when a tryjob fails CQ triggers it again up to a limit). We aim to avoid queuing.

Paweł 
Reply all
Reply to author
Forward
0 new messages