How to check tests in CL for flakiness

[Alexander Yashkin]

Hi all, 

Recently I have tried to upload patch that caused flakiness of tests and was reverted by sheriff.

Locally I was not able to reproduce problem yet I think I fixed the source of flakiness.

Can I somehow run tests for my my CL several times on test machines to ensure they are not flaky, 

or should I simply commit my CL again and wait for result?

WBR Alexander, Yashkin.

[Dirk Pranke]

If you have permission to trigger tryjobs on your own patches, then you can click on the builders you want to run as many times as you like.

If you don't, you can ask your reviewer to do so for you.

-- Dirk

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Paweł Hajdan, Jr.]

+infra-dev,sergiyb,stgao

Curious, do you have more details about that like links to CLs?

Usually flakiness is caused by a race condition. Having good understanding of what the race was and what was done for proper synchronization would be good points for considering this addressed. Checking whether some change happened to fix this, gives less confidence IMO.

Of course having better infra support for testing flakiness could be useful. Adding some people who might be interested. Not sure yet how that fits into bigger picture of infra plans.

Paweł

[Sergiy Byelozyorov]

Hi,

Thank you for sharing your experience. Currently, we do not provide tools to make fixing flakes easier other than the approach suggested by Dirk. We are planning to work on such tools later in the year (probably not before Q3-Q4), but right now our resources are limited and we focus on tools finding which tests are contributing most false rejections and surfacing this information to the developers, e.g. http://chromium-try-flakes.appspot.com/ and http://test-results.appspot.com/flakiness).

We also welcome any contributions and if you are interested, please have a look at some of the ideas that we have or add your own:

• http://crbug.com/601840
  
• http://crbug.com/622359
  

Sergiy

--

Sergiy Byelozyorov | Software Engineer | ser...@google.com

Google Germany GmbH

Erika-Mann-Strasse 33

80636 München

AG Hamburg, HRB 86891 | Sitz der Gesellschaft: Hamburg | Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle

[Alexander Yashkin]

CL is https://codereview.chromium.org/2639153002/.
And with the kind help of vas...@chromium.org I think I understand the
source of test flakiness - its indeed race between loading of
TemplateUrlService and extension reload.

24.01.17 15:09, Paweł Hajdan, Jr. пишет:

> <mailto:chromi...@chromium.org>

> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev
> <http://groups.google.com/a/chromium.org/group/chromium-dev>
>
>
> --
> --
> Chromium Developers mailing list: chromi...@chromium.org

> <mailto:chromi...@chromium.org>

> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev

> <http://groups.google.com/a/chromium.org/group/chromium-dev>
>
>

--
С уважением,
Александр Яшкин,
разработчик Нашего Браузера
http://staff.yandex-team.ru/a-v-y/

[Shuotao Gao]

We had an idea for the verification of fixing flaky tests by a CL. Just captured it down in https://crbug.com/698385, and comments are welcome :)

The general idea is to let CL owners specify which tests they want to run many times (like 100 or even more) in CQ or dry-run, and check for 100% pass rates.

        <mailto:chromium-dev@chromium.org>

        View archives, change email options, or unsubscribe:
        http://groups.google.com/a/chromium.org/group/chromium-dev
        <http://groups.google.com/a/chromium.org/group/chromium-dev>


    --
    --
    Chromium Developers mailing list: chromi...@chromium.org

    <mailto:chromium-dev@chromium.org>

    View archives, change email options, or unsubscribe:
    http://groups.google.com/a/chromium.org/group/chromium-dev
    <http://groups.google.com/a/chromium.org/group/chromium-dev>

--
С уважением,
Александр Яшкин,
разработчик Нашего Браузера
http://staff.yandex-team.ru/a-v-y/

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+unsubscribe@chromium.org.
To post to this group, send email to infr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/d7f85377-7b35-6090-f7d3-192e34e7bcec%40yandex-team.ru.