looking at CQ false rejections

[Paweł Hajdan, Jr.]

I wrote a quick script (attached) to list more details about CQ false rejections, i.e. cases where we failed a CQ attempt for one patchset, but another attempt for the same patchset actually succeeded.

Note this is different from chromium-try-flakes in that the latter does not take into account whether the tryjob failure resulted in the attempt failing or just an internal retry within the same attempt. Of course it's still good to work on these flakes and minimize their occurences since every one increases CQ latency.

Now back to the false rejections, here's the data I got. Note that it only looks at the first page of CQ rejections for now, so the sample size is small.

Still, what I wonder is e.g. whether we should increase the number of retries for blink CQ to 2...

#1: https://chromium-cq-status.appspot.com/patch-status/610963002/1

Try jobs failed on following builders:

  win_blink_rel on tryserver.blink (http://build.chromium.org/p/tryserver.blink/builders/win_blink_rel/builds/29249)

#2: https://chromium-cq-status.appspot.com/patch-status/602373003/20001

Try jobs failed on following builders:

  win_blink_rel on tryserver.blink (http://build.chromium.org/p/tryserver.blink/builders/win_blink_rel/builds/29237)

#3: https://chromium-cq-status.appspot.com/patch-status/610723003/1

Try jobs failed on following builders:

  win_blink_rel on tryserver.blink (http://build.chromium.org/p/tryserver.blink/builders/win_blink_rel/builds/29220)

#4: https://chromium-cq-status.appspot.com/patch-status/612563004/1

Try jobs failed on following builders:

  mac_chromium_rel_swarming on tryserver.chromium.mac (http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_swarming/builds/18283)

#5: https://chromium-cq-status.appspot.com/patch-status/611963002/1

Try jobs failed on following builders:

  android_chromium_gn_compile_rel on tryserver.blink (http://build.chromium.org/p/tryserver.blink/builders/android_chromium_gn_compile_rel/builds/12005)

  mac_blink_compile_dbg on tryserver.blink (http://build.chromium.org/p/tryserver.blink/builders/mac_blink_compile_dbg/builds/20882)

  win_blink_compile_dbg on tryserver.blink (http://build.chromium.org/p/tryserver.blink/builders/win_blink_compile_dbg/builds/21145)

#6: https://chromium-cq-status.appspot.com/patch-status/612713002/1

Try jobs failed on following builders:

  mac_chromium_rel_swarming on tryserver.chromium.mac (http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_swarming/builds/18221)

#7: https://chromium-cq-status.appspot.com/patch-status/609203002/1

Try jobs failed on following builders:

  win_blink_rel on tryserver.blink (http://build.chromium.org/p/tryserver.blink/builders/win_blink_rel/builds/29169)

#8: https://chromium-cq-status.appspot.com/patch-status/609193002/1

Try jobs failed on following builders:

  mac_gpu_triggered_tests on tryserver.chromium.gpu (http://build.chromium.org/p/tryserver.chromium.gpu/builders/mac_gpu_triggered_tests/builds/52328)

#9: https://chromium-cq-status.appspot.com/patch-status/606343002/1

Try jobs failed on following builders:

  win_blink_rel on tryserver.blink (http://build.chromium.org/p/tryserver.blink/builders/win_blink_rel/builds/29157)

#10: https://chromium-cq-status.appspot.com/patch-status/607183003/1

Try jobs failed on following builders:

  win_blink_rel on tryserver.blink (http://build.chromium.org/p/tryserver.blink/builders/win_blink_rel/builds/29083)

Try jobs failed on following builders:

  win_blink_rel on tryserver.blink (http://build.chromium.org/p/tryserver.blink/builders/win_blink_rel/builds/29076)

#11: https://chromium-cq-status.appspot.com/patch-status/607183003/1

Try jobs failed on following builders:

  win_blink_rel on tryserver.blink (http://build.chromium.org/p/tryserver.blink/builders/win_blink_rel/builds/29083)

Try jobs failed on following builders:

  win_blink_rel on tryserver.blink (http://build.chromium.org/p/tryserver.blink/builders/win_blink_rel/builds/29076)

#12: https://chromium-cq-status.appspot.com/patch-status/515813002/60001

Try jobs failed on following builders:

  mac_chromium_rel_swarming on tryserver.chromium.mac (http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_swarming/builds/18018)

#13: https://chromium-cq-status.appspot.com/patch-status/607993002/20001

Try jobs failed on following builders:

  Test-Ubuntu13.10-GCE-NoGPU-x86_64-Debug-Trybot on tryserver.skia (http://108.170.220.120:10117/builders/Test-Ubuntu13.10-GCE-NoGPU-x86_64-Debug-Trybot/builds/891)

#14: https://chromium-cq-status.appspot.com/patch-status/607893003/1

Try jobs failed on following builders:

  win_blink_rel on tryserver.blink (http://build.chromium.org/p/tryserver.blink/builders/win_blink_rel/builds/29017)

Try jobs failed on following builders:

  win_blink_rel on tryserver.blink (http://build.chromium.org/p/tryserver.blink/builders/win_blink_rel/builds/28991)

#15: https://chromium-cq-status.appspot.com/patch-status/591153002/1

Try jobs failed on following builders:

  win_chromium_rel_swarming on tryserver.chromium.win (http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_swarming/builds/16456)

Try jobs failed on following builders:

  chromium_presubmit on tryserver.chromium.linux (http://build.chromium.org/p/tryserver.chromium.linux/builders/chromium_presubmit/builds/12915)

#16: https://chromium-cq-status.appspot.com/patch-status/573353002/60001

Try jobs failed on following builders:

  blink_presubmit on tryserver.blink (http://build.chromium.org/p/tryserver.blink/builders/blink_presubmit/builds/16145)

#17: https://chromium-cq-status.appspot.com/patch-status/607893003/1

Try jobs failed on following builders:

  win_blink_rel on tryserver.blink (http://build.chromium.org/p/tryserver.blink/builders/win_blink_rel/builds/29017)

Try jobs failed on following builders:

  win_blink_rel on tryserver.blink (http://build.chromium.org/p/tryserver.blink/builders/win_blink_rel/builds/28991)

#18: https://chromium-cq-status.appspot.com/patch-status/602193002/40001

Try jobs failed on following builders:

  chromium_presubmit on tryserver.chromium.linux (http://build.chromium.org/p/tryserver.chromium.linux/builders/chromium_presubmit/builds/13781)

#19: https://chromium-cq-status.appspot.com/patch-status/603683008/1

Try jobs failed on following builders:

  chromium_presubmit on tryserver.chromium.linux (http://build.chromium.org/p/tryserver.chromium.linux/builders/chromium_presubmit/builds/13768)

#20: https://chromium-cq-status.appspot.com/patch-status/608713002/20001

Try jobs failed on following builders:

  win_blink_rel on tryserver.blink (http://build.chromium.org/p/tryserver.blink/builders/win_blink_rel/builds/28950)

#21: https://chromium-cq-status.appspot.com/patch-status/586753002/80001

Try jobs failed on following builders:

  win_blink_rel on tryserver.blink (http://build.chromium.org/p/tryserver.blink/builders/win_blink_rel/builds/28948)

#22: https://chromium-cq-status.appspot.com/patch-status/598363003/100001

Try jobs failed on following builders:

  chromium_presubmit on tryserver.chromium.linux (http://build.chromium.org/p/tryserver.chromium.linux/builders/chromium_presubmit/builds/13755)

#23: https://chromium-cq-status.appspot.com/patch-status/594843004/20001

Try jobs failed on following builders:

  win_blink_rel on tryserver.blink (http://build.chromium.org/p/tryserver.blink/builders/win_blink_rel/builds/28908)

#24: https://chromium-cq-status.appspot.com/patch-status/600163004/60001

Try jobs failed on following builders:

  linux_chromium_chromeos_rel_swarming on tryserver.chromium.linux (http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_rel_swarming/builds/18199)

#25: https://chromium-cq-status.appspot.com/patch-status/601363004/1

Try jobs failed on following builders:

  win_blink_rel on tryserver.blink (http://build.chromium.org/p/tryserver.blink/builders/win_blink_rel/builds/28895)

#26: https://chromium-cq-status.appspot.com/patch-status/602013002/80001

Try jobs failed on following builders:

  chromium_presubmit on tryserver.chromium.linux (http://build.chromium.org/p/tryserver.chromium.linux/builders/chromium_presubmit/builds/13716)

Paweł

[John Abd-El-Malek]

I can't help but notice that most of the links below are from win_blink_rel. Looking at Sergey's weekly emails to chromium-dev, there's a section about top flaky builders. I ran the numbers for last week. win_blink_rel was flaky 26% of the time. win_blink_dbg was 22%. The next blink bot was only at 3%. So something is much more flakier in the blink win bots.

For comparison, on the chromium CQ, the two win bots (32 & 64 bit) are at 3%. Note that when we started the CY, the win_chromium bots were around 20%. We got this decrease mostly by disabling the very small number of tests that were responsible for most of the flakiness. Some of the disabled tests were fixed, but the important thing is that they were disabled immediately. chromium-try-flakes has helped me to get this list.

Instead of making the CQ retry twice for blink and wasting cycles rerunning win_blink_rel so much, it seems that whatever is running on that bot should be examined closely to bring the flakiness rate from a very high 30% down to something reasonable.

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To post to this group, send email to infr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAATLsPbhQC%3DgLjdHa4cJ2t_WH9miL4ERVEq30xow2ZaVqUSOOQ%40mail.gmail.com.

[Dirk Pranke]

win_blink_rel is indeed known to be very flaky. enne@ landed some changes at the end of the week last week that I'm hoping fix the worst of the problems, but I haven't looked at recent builds to see if things are better.

Unfortunately the intersection of {blink developers} and {people who regularly develop on windows} is nearly zero, and the non-zero few are usually quite busy, so there are very few people regularly feeling this pain *and* able and motivated to fix it.

Volunteers to work on issues are welcome :). We could theoretically be willing to turn the bot off as an alternative, but I'm not sure how that would be helpful.

-- Dirk

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CALhVsw34wQrF%2BY8SsxiA2DX-yp1Rm-ZQzqtG0VOXMaLUp5nt2g%40mail.gmail.com.

[John Abd-El-Malek]

On Mon, Sep 29, 2014 at 1:42 PM, Dirk Pranke <dpr...@chromium.org> wrote:

win_blink_rel is indeed known to be very flaky. enne@ landed some changes at the end of the week last week that I'm hoping fix the worst of the problems, but I haven't looked at recent builds to see if things are better.

Unfortunately the intersection of {blink developers} and {people who regularly develop on windows} is nearly zero, and the non-zero few are usually quite busy, so there are very few people regularly feeling this pain *and* able and motivated to fix it.

nit: everyone working on Blink is feeling the pain, since it appears the high flakiness is slowing down the Blink CQ by 20-30 minutes :)

[Dirk Pranke]

Which is why I emphasized the "and" part ...

[Julie Parent]

+eae

Not volunteering him, but Emil is the only person I know who falls in the intersection of {blink developers} and {people who regularly develop on windows}, and has indicated willingness to help with CY.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAEoffTCvF25pwqoLbihKVX%3Doy8LyzWeQf2r8yKMgSRqP_hfXCg%40mail.gmail.com.

[Emil A Eklund]

On Mon, Sep 29, 2014 at 4:32 PM, Julie Parent <jpa...@chromium.org> wrote:
> +eae
>
> Not volunteering him, but Emil is the only person I know who falls in the
> intersection of {blink developers} and {people who regularly develop on
> windows}, and has indicated willingness to help with CY.

Sad but true, that intersection is surprisingly small given that
windows is still by far our most popular platform.

What specifically am I (not) being volunteered for?

--
Emil

[Julie Parent]

Getting the 26% flaky rate of win_blink tests down to something more reasonable, by investigating and disabling tests as necessary, like jam@ did with chromium tests.  https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CALhVsw34wQrF%2BY8SsxiA2DX-yp1Rm-ZQzqtG0VOXMaLUp5nt2g%40mail.gmail.com should have the full context

[Ojan Vafai]

IMO, the focus here should be on exposing flakiness on the main waterfall bots in sheriff-o-matic so we can disable/fix/delete/etc these tests in a long-term sustainable way. I have a plan for this, but I've been having trouble finding someone to work on this.

That said, it'd probably be good for someone to check a random sample of the false rejections we see on the win blink try bot and see if they're flaky on the main waterfall.

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAPSmAATLMunerXT%2BBTZHau4QV%3DAPZRK81FHS9Dkf-NSZtf%3DCng%40mail.gmail.com.

[Emil A Eklund]

On Tue, Sep 30, 2014 at 5:40 PM, Julie Parent <jpa...@chromium.org> wrote:
> Getting the 26% flaky rate of win_blink tests down to something more
> reasonable, by investigating and disabling tests as necessary, like jam@ did
> with chromium tests.
> https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CALhVsw34wQrF%2BY8SsxiA2DX-yp1Rm-ZQzqtG0VOXMaLUp5nt2g%40mail.gmail.com
> should have the full context

Ah, that certainly sounds like something I'd be able to help with.

[Ojan Vafai]

To clarify my previous comment, if you're willing to work on actually fixing the flakiness on windows, that's totally crucial and you should do that. Anyone on the team is well qualified to do the sheriff-o-matic work I mentioned, but you're one of the few well-qualified to fix the flakiness.

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CADu_oUAh9PgfwTmy6VVjMPYS_HDzpCJX9%2BMztw6VkNbEO70_Jw%40mail.gmail.com.

[John Abd-El-Malek]

Is it known if it's a number of flaky tests vs something structural that affects all tests?

If the former, I had been blacklisting blink tryjobs from chromium-try-flakes. Enabling it is is trivial (removing the if statement in line 273). This would show which tests fail and pass in different tryjobs for the same patchset.

[Dirk Pranke]

I've not seen anything to think that there is structural flakiness that affects *all* tests.

I think there are bugs that affect certain subsets of tests. I don't think we know exactly what they all are, but the bug that enne fixed last week was a good example of such things.

I think the current level of flakiness is an example of what happens when the tools get good enough so that you can ignore things without feeling the pain: if no one is inclined to run the tests locally, and things eventually pass in a couple hours, it's easy enough to ignore the problem (i.e., I think there are qualitative differences between try jobs that complete in an hour and jobs that complete in ten hours).

-- Dirk

[Ojan Vafai]

Historically, the http tests on windows have been very flaky and that's clearly something structural. But I haven't looked recently to see if that's still the case.

[Dirk Pranke]

I believe switching to apache fixed that; I haven't seen any issues the past few months.

-- Dirk

[John Abd-El-Malek]

ok, I've made chromium-try-flakes start showing blink try flakes. We'll see what data it shows by the morning.

[Ilya Tikhonovsky]

I ran a script against win_blin_rel try bot logs  and got the next stats for 200 runs (only failures were counted)

name	total	text	timeouts	crashes	image	missing
media/encrypted-media/encrypted-media-playback-multiple-sessions.html	38	38	0	0	0	0
virtual/antialiasedtext/fast/text/orientation-sideways.html	27	0	0	0	27	0
fast/writing-mode/english-lr-text.html	27	0	0	0	27	0
virtual/antialiasedtext/fast/text/international/vertical-text-glyph-test.html	27	0	0	0	27	0
virtual/antialiasedtext/fast/text/decorations-with-text-combine.html	27	0	0	0	27	0
virtual/antialiasedtext/fast/text/justify-ideograph-vertical.html	27	0	0	0	27	0
virtual/antialiasedtext/fast/text/international/text-combine-image-test.html	27	0	0	0	27	0
fast/css/font-weight-1.html	27	0	0	0	27	0
http/tests/w3c/webperf/submission/Intel/user-timing/test_user_timing_measure_associate_with_navigation_timing.html	23	23	0	0	0	0
inspector/tracing/timeline-receive-response-event.html	20	20	0	0	0	0
fast/pagination/div-x-horizontal-bt-ltr.html	20	0	0	20	0	0
virtual/deferred/inspector/tracing/timeline-receive-response-event.html	20	20	0	0	0	0
virtual/implsidepainting/inspector/tracing/timeline-receive-response-event.html	20	20	0	0	0	0
http/tests/media/media-source/mediasource-play-then-seek-back.html	18	18	0	0	0	0
fast/css/fontfaceset-add-remove-while-loading.html	16	16	0	0	0	0
fast/pagination/div-x-horizontal-bt-rtl.html	15	0	0	15	0	0
fast/multicol/newmulticol/compare-with-old-impl/div-x-horizontal-bt-ltr.html	14	0	0	14	0	0
fast/multicol/newmulticol/compare-with-old-impl/div-x-horizontal-bt-rtl.html	11	0	0	11	0	0
virtual/regionbasedmulticol/fast/pagination/div-x-horizontal-bt-ltr.html	8	0	0	8	0	0
media/encrypted-media/encrypted-media-needkey.html	7	7	0	0	0	0

.......

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CALhVsw2Fi01jaY87sGgWV%2BuZy_x6OQhMpp6KUi5uMg9EDKg8sg%40mail.gmail.com.

[Ojan Vafai]

This list is just failures, not flakes, right? 

[Ilya Tikhonovsky]

yep

the table of flakes is quite different

name	total	flaky text	flaky timeouts	flaky crashes	flaky image
virtual/gpu/fast/canvas/check-stale-putImageData.html	175	0	0	175	0
battery-status/page-visibility.html	128	0	0	128	0
http/tests/plugins/interrupted-get-url.html	71	0	71	0	0
http/tests/appcache/offline-access.html	67	0	67	0	0
http/tests/appcache/video.html	54	0	54	0	0
http/tests/security/link-crossorigin-subresource-use-credentials.html	36	0	36	0	0
http/tests/inspector/extensions-ignore-cache.html	34	0	34	0	0
http/tests/security/cross-frame-access-frameelement.html	30	0	30	0	0
http/tests/security/img-crossorigin-no-credentials-prompt.html	26	0	26	0	0
http/tests/security/mime-type-execute-as-html-16.html	24	0	24	0	0
http/tests/pointer-lock/pointerlockelement-different-origin.html	21	0	21	0	0
http/tests/security/script-onerror-crossorigin-same-origin.html	21	0	21	0	0
http/tests/w3c/webperf/approved/UserTiming/test_user_timing_mark.htm

18

18

0

0

0

http/tests/misc/dns-prefetch-control.html	18	0	18	0	0
http/tests/w3c/webperf/approved/UserTiming/test_user_timing_measure.htm

18

18

0

0

0

http/tests/security/cross-frame-access-protocol-explicit-domain.html	18	0	18	0	0
http/tests/inspector/extensions-useragent.html	17	0	17	0	0
fast/text/ellipsis-stroked.html	16	0	0	0	16
http/tests/security/referrer-policy-conflicting-policies.html	16	0	16	0	0
http/tests/https/verify-ssl-enabled.php	16	0	16	0	0
editing/pasteboard/4944770-2.html

15

0

0

15

0

fast/forms/implicit-submission.html

14

0

0

14

0

http/tests/htmlimports/redirect-cross-origin-cross-same.html	14	0	14	0	0
http/tests/appcache/non-html.xhtml	14	0	14	0	0
http/tests/media/video-cancel-load.html	13	0	13	0	0
inspector/sources/debugger/debugger-pause-on-blocked-event-handler.html	13	0	0	13	0
http/tests/security/script-crossorigin-loads-correctly-credentials-2.html	12	0	12	0	0
fast/multicol/hit-test-gap-between-pages-flipped.html

11

0

0

11

0

http/tests/incremental/doc-write-before-end.pl	10	0	10	0	0
http/tests/security/img-with-failed-cors-check-fails-to-load.html	10	0	10	0	0
http/tests/media/video-buffered.html	10	0	10	0	0
http/tests/loading/preload-picture-sizes-2x.html	10	0	10	0	0
inspector-enabled/sources/debugger/script-window-close-breakpoint.html	10	0	0	10	0
fast/overflow/overflow-rtl-vertical-origin.html	10	0	0	10	0
http/tests/media/remove-while-loading.html	9	0	9	0	0
media/track/track-cue-nothing-to-render.html	9	9	0	0	0
css3/flexbox/flexbox-overflow-auto.html	9	0	0	9	0
media/encrypted-media/encrypted-media-playback-multiple-sessions.html	9	6	0	3	0
http/tests/media/reload-after-dialog.html	8	0	8	0	0
http/tests/inspector/network-preflight-options.html	8	0	8	0	0
media/media-fragments/TC0037.html	8	8	0	0	0
fast/multicol/newmulticol/compare-with-old-impl/div-x-horizontal-bt-rtl.html

8

0

0

8

0

inspector/editor/text-editor-word-jumps.html	8	0	8	0	0
fast/mediastream/RTCPeerConnection-statsSelector.html	8	8	0	0	0
http/tests/inspector/elements/styles/stylesheet-tracking.html

8

0

0

8

0

http/tests/security/script-with-failed-cors-check-fails-to-load.html	8	0	8	0	0
inspector-protocol/cpu-profiler/console-profile.html

8

0

0

8

0

inspector-protocol/loading-iframe-document-node.html

8

0

0

8

0

imported/web-platform-tests/custom-elements/creating-and-passing-registries/share-registry-import-document.html	7	0	7	0	0
media/media-fragments/TC0006.html

7

7

0

0

0

media/media-fragments/TC0024.html

7

7

0

0

0

http/tests/security/text-track-crossorigin.html	7	0	7	0	0
fast/pagination/div-x-horizontal-bt-ltr.html	7	0	0	7	0
http/tests/security/window-events-clear-domain.html	7	0	7	0	0
fast/dom/partial-layout-non-overlay-scrollbars.html

7

7

0

0

0

inspector/tracing/timeline-bound-function.html	7	0	7	0	0
http/tests/inspector/resource-har-conversion.html	7	0	7	0	0
fast/mediastream/RTCPeerConnection-stats.html

7

7

0

0

0

inspector/timeline/timeline-bound-function.html	7	0	7	0	0
http/tests/security/host-compare-case-insensitive.html	7	0	7	0	0
http/tests/media/video-buffered-range-contains-currentTime.html	6	0	6	0	0
media/track/track-css-matching-timestamps.html	6	6	0	0	0
http/tests/plugins/cross-frame-object-access.html	6	1	5	0	0
fast/css/fontfaceset-add-remove-while-loading.html	6	6	0	0	0
http/tests/media/media-source/mediasource-play-then-seek-back.html	6	6	0	0	0
http/tests/security/local-video-source-from-remote.html	6	0	6	0	0
http/tests/loading/preload-picture-sizes.html	6	0	6	0	0
http/tests/misc/selectionAsMarkup.html	6	0	6	0	0
media/track/track-css-matching.html	6	4	2	0	0
svg/custom/resource-client-removal.svg	6	6	0	0	0
http/tests/media/text-served-as-text.html	6	0	6	0	0
media/media-fragments/TC0039.html	6	6	0	0	0
http/tests/security/cross-frame-access-set-window-properties.html	6	0	6	0	0
fast/dom/HTMLImageElement/image-srcset-w-onerror.html	6	6	0	0	0
inspector/elements/styles/styles-add-blank-property.html	6	6	0	0	0
http/tests/loading/dont-preload-non-img-srcset.html	6	0	6	0	0
media/track/track-cues-seeking.html	6	6	0	0	0
printing/ellipsis-printing-style.html	6	0	0	0	6

[Eric Seidel]

You might be able to find help by roping in blink-dev.  Anyone can triage flaky test lists and disable tests.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAD74ZUBp%3DPvzqKnL46dg%2BAX89tddHWvEJNPW%3DtV6APZwd0e2wg%40mail.gmail.com.

[Ilya Tikhonovsky]

I created a small page which fetches stdout from try bot runs and shows the list of flaky and failing tests.
https://x20web.corp.google.com/~loislo/try_bot_flakiness.html

I think it needs to be converted/replaced with flakiness dashboard because it is impossible to detect the type of failure from the webkit_test step stdout and it takes too much time to get the data from the full stdout. But try bots don't publish the results to test-results server at the moment.

It seems that about a half of the flaky tests on windows are from inspector/tracing/ tests.

I've marked them as flaky and found && fixed one of the reason of the flakiness.

BTW: It is interesting that some tests always fail/crash/timeout in the batch and always pass on retry. So they look almost always green in test-results server.

[Dirk Pranke]

On Tue, Oct 7, 2014 at 11:31 AM, Ilya Tikhonovsky <loi...@google.com> wrote:

I created a small page which fetches stdout from try bot runs and shows the list of flaky and failing tests.
https://x20web.corp.google.com/~loislo/try_bot_flakiness.html

I think it needs to be converted/replaced with flakiness dashboard because it is impossible to detect the type of failure from the webkit_test step stdout and it takes too much time to get the data from the full stdout. But try bots don't publish the results to test-results server at the moment.

The try jobs do publish their results to google storage, so you could download them from there. 

It seems that about a half of the flaky tests on windows are from inspector/tracing/ tests.

I've marked them as flaky and found && fixed one of the reason of the flakiness.

Yup, that matches what I saw on Friday. Good to hear it's been addressed.

 

BTW: It is interesting that some tests always fail/crash/timeout in the batch and always pass on retry. So they look almost always green in test-results server.

That's actually the exact opposite of what should be happening. If a test fails initially and passes on the retry, then test-results should show the failure, not the pass; put differently, test-results ignores the retries completely. 

If that's not what we're seeing, either that's a bug or someone changed how test-results works for the worse :).

-- Dirk