How to (locally) test browser tests for flakiness?

[Daniel Cheng]

I have some browser tests that are flaky. I try to repro the flakiness with a command like this:

testing/xvfb.py out/release/content_browsertests --gtest_filter=MyBrowserTests.\* --gtest_repeat=1000 --test-launcher-bot-mode

Unfortunately, at the end, it doesn't actually print out that any iterations failed. It just says this:

Summary of all test iterations:

End of the summary.

I have to stare at the tests the entire time they are running to see that some failed. Is it possible to get the test launcher to print out if some repeats failed?

Daniel

[Dave Tapuska]

I usually add gtest_break_on_failure as well. Then it stops on the first failure. 

Dave

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAF3XrKoDDyO-D5_si67A_PtfLc6sRUr-Wek-81wU%2ByS_OVezmA%40mail.gmail.com.

[Daniel Cheng]

Hm... this is a useful flag to know about, but at least for the browser tests I'm investigating, it greatly slows down execution speed (and parallelism and hence, flakiness)—I'm guessing it disables whatever --test-launcher-bot-mode does.

Daniel

[Ben Pastene]

I'm not sure, but I imagine there's a cmd-line switch to make the runner do what you want.

Alternatively, if you want the results rendered in the web UI the same way they are on the bots, you can do a couple easy-ish things:

• Install the result_adapter tool from CIPD.
• ie: run `cipd install 'infra/tools/result_adapter/${platform}'`
• Append the test invocation with the output json flag, and wrap the whole thing in a couple more commands.
• ie: `rdb stream -new -realm chromium:public -- ../result_adapter gtest -result-file out.json -- testing/xvfb.py out/release/content_browsertests --gtest_filter=MyBrowserTests.\* --gtest_repeat=1000 --test-launcher-bot-mode --test-launcher-summary-output=out.json`

At the end it should print a link to luci-milo.appspot.com that displays the results of the run, eg: https://luci-milo.appspot.com/ui/inv/u-bpastene-2022-11-08-17-19-00-ec7fb8fef95df91f

Any test with a failure should show up as "flaky" in that UI. And you should be able to expand each test to see the total amount of failures (might require some scrolling though). So maybe that can get you what you need?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAF3XrKqYGe3AZctgdk%3D8DrmYXm1NVHfV4TdYt8ay4wHYad%3DyDA%40mail.gmail.com.

[Christoph Schwering]

Does --gtest_break_on_failure help? It makes execution stop after the first failure.

--

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAF3XrKoDDyO-D5_si67A_PtfLc6sRUr-Wek-81wU%2ByS_OVezmA%40mail.gmail.com.

--

Google Germany GmbH

Erika-Mann-Straße 33

80636 München

Geschäftsführer: Paul Manicle, Liana Sebastian

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls Sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde. 

This email is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.