PSA: new test launcher is here

[Paweł Hajdan, Jr.]

The bots are now running with --brave-new-test-launcher, the new C++ test launcher that is intended to be more solid and provide more detailed report from each test run that can be analyzed later. It is also faster by running tests in parallel.

This is not really new - an initial PSA has been sent two months ago: https://groups.google.com/a/chromium.org/d/msg/chromium-dev/jej3uEF1WJ4/Hp1E6BAmWq4J

I appreciate your feedback and I've strived to apply it best I could so far. If I missed something, please file a bug assigned directly to me so it doesn't get lost, and it's also OK to ping me randomly for any questions you might have. You can also send me an e-mail or just reply to this thread.

I'd like to highlight some command-line switches that may be very useful to you:

--test-launcher-developer-mode: enable this if you're debugging a failing test locally; it disables test retries and output buffering

--test-launcher-jobs: controls number of parallel jobs; use --test-launcher-jobs=1 for serial execution

--test-launcher-print-test-stdio: can take "auto" (default), "always", or "never" values. By default the launcher prints test output snippet only on failures. "always" makes it print the snippet always and so on.

Note that I'm open to ideas and CLs to change these options or add new ones. I'll try to keep the number of knobs reasonable, and will challenge the change ideas to provide convincing rationale and data as needed. Still, this is a launcher done by a developer, for developers. I'd like it to be helpful and useful, so please keep the feedback coming.

I also think we're getting ready to enable it by default (as opposed to having it behind a flag; bots are running it now anyway). Please do your tests, see how it works, and let me know what you think.

Finally, if your test doesn't react to --brave-new-test-launcher, please file a bug for me. I can convert test targets for you, it's easy, it's mostly about enumerating ones that still need it. Help obviously welcome.

Paweł

[Nico Weber]

Congrats!

Why wasn't switching the bots done by turning it on by default?

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Gabriel Charette]

Man... mobile gmail really fails at multi-account.. and I apparently fail at realizing it... from right accoun this time... again...

On Tue, Nov 12, 2013 at 9:55 AM, Gabriel Charette <g...@google.com> wrote:

On Nov 11, 2013 9:52 PM, "Paweł Hajdan, Jr." <phajd...@chromium.org> wrote:
>
> The bots are now running with --brave-new-test-launcher, the new C++ test launcher that is intended to be more solid and provide more detailed report from each test run that can be analyzed later. It is also faster by running tests in parallel.
>
> This is not really new - an initial PSA has been sent two months ago: https://groups.google.com/a/chromium.org/d/msg/chromium-dev/jej3uEF1WJ4/Hp1E6BAmWq4J
>
> I appreciate your feedback and I've strived to apply it best I could so far. If I missed something, please file a bug assigned directly to me so it doesn't get lost, and it's also OK to ping me randomly for any questions you might have. You can also send me an e-mail or just reply to this thread.
>
> I'd like to highlight some command-line switches that may be very useful to you:
>
> --test-launcher-developer-mode: enable this if you're debugging a failing test locally; it disables test retries and output buffering

Feels this should be default and the bots should instead specify a flag to disable developer mode.

In general I think it's better to have a more complex config for the bots when it avoids requiring developers to remember specific flags required for debugging.

Cheers!
Gab

>
> --test-launcher-jobs: controls number of parallel jobs; use --test-launcher-jobs=1 for serial execution
>
> --test-launcher-print-test-stdio: can take "auto" (default), "always", or "never" values. By default the launcher prints test output snippet only on failures. "always" makes it print the snippet always and so on.
>
> Note that I'm open to ideas and CLs to change these options or add new ones. I'll try to keep the number of knobs reasonable, and will challenge the change ideas to provide convincing rationale and data as needed. Still, this is a launcher done by a developer, for developers. I'd like it to be helpful and useful, so please keep the feedback coming.
>
> I also think we're getting ready to enable it by default (as opposed to having it behind a flag; bots are running it now anyway). Please do your tests, see how it works, and let me know what you think.
>
> Finally, if your test doesn't react to --brave-new-test-launcher, please file a bug for me. I can convert test targets for you, it's easy, it's mostly about enumerating ones that still need it. Help obviously welcome.
>
> Paweł
>

[Dana Jansens]

Is it meant to be possible to combine --brave-new-test-launcher with --gtest_filter? I'm finding that it is ignoring the filter when I use the brave new launcher.

--

[Adrienne Walker]

At the moment, it seems you have to pass --single-process-tests to use
--gtest_filter.

2013/11/12 Dana Jansens <dan...@chromium.org>:

[Paweł Hajdan, Jr.]

To clear up possible confusion: you're most likely running cc_unittests. This is https://code.google.com/p/chromium/issues/detail?id=299200 . Please let me know if any other test targets are affected.

For info why test output buffering works the way it does and why the flags are designed in this way, please see https://groups.google.com/a/chromium.org/d/msg/chromium-dev/Mw1hrGwnUpw/czyRDIB8juoJ and my comments on https://codereview.chromium.org/61363002 .

About deployment strategy: there are multiple reasons. One is not doing too much at once. As you see there are some developer-related questions to resolve that don't really affect bots. I'm aiming at resolving them before the new behavior gets enabled by default. Then enabling it on the bots was done in stages, to both support backward compatibility with branches (dev, beta, stable - they obviously have different code from trunk whereas some scripts like runtest.py are common for all of them) and removing some intermediaries/shims like sharding_supervisor.py. With the new test launcher many changes will become easy to test via trybots - and I'll be moving even more parts of the test infrastructure in that direction.

Please keep the questions/feedback coming, I'd like to identify as many issues beforehand as possible. Especially weirdly behaving test targets, or one that don't yet support the new test launcher.

Paweł

[Dana Jansens]

On Tue, Nov 12, 2013 at 2:58 PM, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

To clear up possible confusion: you're most likely running cc_unittests. This is https://code.google.com/p/chromium/issues/detail?id=299200 . Please let me know if any other test targets are affected.

I am, thanks. Will do.

[Paweł Hajdan, Jr.]

Thank you. By the way I'll indeed change the defaults so that developers don't need to remember crazy flags, see https://groups.google.com/a/chromium.org/d/msg/chromium-dev/Mw1hrGwnUpw/mOdN8cV2poYJ . In fact, several changes are in review now, it'll happen this week.

Paweł

[Paweł Hajdan, Jr.]

Due to some internal waterfalls weirdness this was delayed (that was the only issue, and it couldn't be predicted beforehand without duplicating all of the waterfalls, which is obviously not feasible).

New defaults are enabled now: when you use --brave-new-test-launcher it defaults to developer mode unless --test-launcher-bot-mode is present.

Please let me know if you'd like to see further adjustments, I'm quite open to that. I'd like to make --brave-new-test-launcher itself default ~next week (there'll still be option to use plain gtest just in case).

Paweł

[James Robinson]

On Wed, Nov 27, 2013 at 11:32 AM, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

Due to some internal waterfalls weirdness this was delayed (that was the only issue, and it couldn't be predicted beforehand without duplicating all of the waterfalls, which is obviously not feasible).

New defaults are enabled now: when you use --brave-new-test-launcher it defaults to developer mode unless --test-launcher-bot-mode is present.

 

Please let me know if you'd like to see further adjustments, I'm quite open to that. I'd like to make --brave-new-test-launcher itself default ~next week (there'll still be option to use plain gtest just in case).

Can you make --help document the available flags and what they do?

- James 

[Lei Zhang]

I already filed http://crbug.com/322266 for this.

[Ilya Sherman]

Can --test-launcher-bot-mode be shortened to something more like --quiet --parallel?  I'd rather have more simple flags than one complex flag, as they're easier to understand (within reason, of course).

[Paweł Hajdan, Jr.]

On Wed, Nov 27, 2013 at 2:33 PM, Ilya Sherman <ishe...@chromium.org> wrote:

Can --test-launcher-bot-mode be shortened to something more like --quiet --parallel?  I'd rather have more simple flags than one complex flag, as they're easier to understand (within reason, of course).

This is easy to respond to. I don't think that'd be a good idea.

Changing these flags build-system side first requires updating multiple places, and depending on the exact place may either require a master restart, would be impossible to test using trybots, or both.

Furthermore, it's not always --parallel since interactive_ui_tests run in serial mode (single job). It's not only --quiet but also enables test retries. Next step is likely uploading the JSON summaries to Google Storage so they can be analyzed with Big Query. Should each of these be a flag, with all the costs and ambiguities?

The main idea behind it is we'd configure the bot environment to pass that flag once. The interpretation of that lives in src, which can be tested via trybots, can be branched, and has working blamelists.

To give some graphic example of why changing flags buildbot-side is a bad idea: it introduced this weird 2-week delay in landing an otherwise very simple change. It made other changes unnecessarily risky and hard to test, because the test would require duplicating all waterfalls, which we can't do.

I think I see the reasons behind the question, and I agree about understandability. More specific suggestions taking the above into account are welcome. One possible idea would be to make sure --test-launcher-bot-mode can be expressed as a set of other existing flags and make the test launcher print these flags.

Paweł

[Ilya Sherman]

On Wed, Nov 27, 2013 at 2:51 PM, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

On Wed, Nov 27, 2013 at 2:33 PM, Ilya Sherman <ishe...@chromium.org> wrote:

Can --test-launcher-bot-mode be shortened to something more like --quiet --parallel?  I'd rather have more simple flags than one complex flag, as they're easier to understand (within reason, of course).

This is easy to respond to. I don't think that'd be a good idea.

Changing these flags build-system side first requires updating multiple places, and depending on the exact place may either require a master restart, would be impossible to test using trybots, or both.

Furthermore, it's not always --parallel since interactive_ui_tests run in serial mode (single job). It's not only --quiet but also enables test retries. Next step is likely uploading the JSON summaries to Google Storage so they can be analyzed with Big Query. Should each of these be a flag, with all the costs and ambiguities?

For the flag used on the bots, that seems fair enough.  On the other hand, developers probably don't want JSON summaries to be uploaded to Google Storage from their local test runs.  I'm mostly concerned about the flags that developers use, rather than what bots use.

The main idea behind it is we'd configure the bot environment to pass that flag once. The interpretation of that lives in src, which can be tested via trybots, can be branched, and has working blamelists.

To give some graphic example of why changing flags buildbot-side is a bad idea: it introduced this weird 2-week delay in landing an otherwise very simple change. It made other changes unnecessarily risky and hard to test, because the test would require duplicating all waterfalls, which we can't do. 

 

I think I see the reasons behind the question, and I agree about understandability. More specific suggestions taking the above into account are welcome. One possible idea would be to make sure --test-launcher-bot-mode can be expressed as a set of other existing flags and make the test launcher print these flags.

That seems reasonable to me.  That both makes it possible for developers to enable just the parts that they care about when running the tests locally, and helps understand failures based on logs from the bots.

[Chris Hopman]

It's too bad that changing bot configurations is so involved.

--

[Paweł Hajdan, Jr.]

I'm risking some major bikeshed, but I uploaded https://codereview.chromium.org/103413003/ to update --help output.

Note that for unit tests you'd need to pass --brave-new-test-launcher --help to see that help instead of gtest's help. gtest's help is still available as --gtest_help.

Paweł

[Elliott Sprehn]

Does the flags library in base not support automatic --help based on what flags are declared? Coming from google internal projects it's really surprising to see hard coded help like that where the flags can become out of sync with what's actually available.

[Paweł Hajdan, Jr.]

Please see the following references for some background why it works the way it works in Chrome:

https://code.google.com/p/chromium/issues/detail?id=26821

https://groups.google.com/a/chromium.org/d/msg/chromium-dev/jZmIKnpfhx8/ZBV4N4538wIJ

https://groups.google.com/a/chromium.org/d/msg/chromium-dev/k6-gDw67gIE/uHIgM5PQW1wJ

Also note that supported/effective flags change depending on the launcher (unit vs. browser tests), but since for browser tests both launchers are linked (but only one is used) auto-generated --help wouldn't be correct.

Paweł

[Paweł Hajdan, Jr.]

New test launcher is now default, also when not running on the bots (e.g. developer machines).

You can pass --help flag to the test binary to see what flags it supports (note that it changed since my previous e-mail).

Thank you for your feedback that made this possible, and for patience with some breakages on less common configurations. Any further changes will be now much easier to test - just using trybots. Previously this was not possible, as e.g. runtest.py file is used on all the bots and there is no tryserver for it. Another advantage is you're now running the same code that the bots are running - which should result in better consistency and less surprises.

Please keep the feedback coming - and if you made a comment I didn't reply to or didn't implement please let me know - it could just get lost when I was fixing the breakages.

Here's an example of unit test binary --help output (note that browser tests, e.g. content_browsertests, browser_tests, interactive_ui_tests have a slightly different set of flags - this is reflected in their --help):

Runs tests using the gtest framework, each batch of tests being
run in their own process. Supported command-line flags:
 Common flags:
  --gtest_filter=...
    Runs a subset of tests (see --gtest_help for more info).
  --help
    Shows this message.
  --gtest_help
    Shows the gtest help message.
  --test-launcher-jobs=N
    Sets the number of parallel test jobs to N.
  --single-process-tests
    Runs the tests and the launcher in the same process. Useful
    for debugging a specific test in a debugger.
 Other flags:
  --test-launcher-batch-limit=N
    Sets the limit of test batch to run in a single process to N.
  --test-launcher-retry-limit=N
    Sets the limit of test retries on failures to N.
  --test-launcher-summary-output=PATH
    Saves a JSON machine-readable summary of the run.
  --test-launcher-print-test-stdio=auto|always|never
    Controls when full test output is printed.
    auto means to print it when the test failed.
  --test-launcher-total-shards=N
    Sets the total number of shards to N.
  --test-launcher-shard-index=N
    Sets the shard index to run to N (from 0 to TOTAL - 1).

Paweł

[Vincent Scheib]

When I want to run locally as fast as possible, the easiest route is to use --test-launcher-bot-mode, right? If so, it seems that it should be listed in the help options so I don't have to make a note of it elsewhere.

[Antony Sargent]

I just ran into the problem that unit_tests claims to no longer support --gtest_list_tests, and found a "// TODO(phajdan.jr): Restore support for the flag." in the code where the error message comes from. I did a little digging and found that I can actually get it to work by running "unit_tests --single-process-tests --gtest_list_test", but it would be nice to get this fixed. Pawel - is there already a bug on file for this?

[Paweł Hajdan, Jr.]

I uploaded https://codereview.chromium.org/118723005/ to restore --gtest_list_tests .

Paweł

[Paweł Hajdan, Jr.]

Bot mode might do some more things in the future.

It's not obvious to me what would work really well for people, given that many of my initial guesses were wrong.

Do you have some suggestions?

Paweł

[Vincent Scheib]

:\ I think bot mode makes sense for the bots.

I'm guessing there's roughly 2 main developer workflows:

- Iterating on one or a small number of tests, desiring verbose logging, serial execution, etc.

- Running a larger batch of tests as fast as possible, with parallel and other options being set automatically if possible. 

I.e., I'd love to just remember to either run with no flags, or with one simple flag e.g. --auto-parallel. 

[Shadi Khalek]

I noticed that browser_tests now show a black screen when running tests, for example:

out/Release/browser_tests --gtest_filter=MSE_ClearKey/EncryptedMediaTest.Playback_VideoOnly_WebM/0

Does anyone know if that's controlled by the new test launcher flags or related to something else? 

[Dana Jansens]

On Wed, Mar 5, 2014 at 2:13 PM, Shadi Khalek <sh...@chromium.org> wrote:

I noticed that browser_tests now show a black screen when running tests, for example:

out/Release/browser_tests --gtest_filter=MSE_ClearKey/EncryptedMediaTest.Playback_VideoOnly_WebM/0

Does anyone know if that's controlled by the new test launcher flags or related to something else? 

The --enable-pixel-output-in-tests flag will cause them to draw. See also: https://groups.google.com/a/chromium.org/forum/#!search/enable-pixel-output-in-tests$20chromium-dev/chromium-dev/WACWiwFA0Rk/ZneYb_jvFroJ

[oshima]

Hi,

I noticed that interactive_ui_tests are running with --brave-new-test-launcher on bots. I thought this test cannot

be parallelized because it relies on actual mouse position and/or native window activation.

Am I missing something? Or has this problem been solved somehow?

Thanks,

- oshima

[Paweł Hajdan, Jr.]

The knowledge of that is now embedded into the binary - interactive_ui_tests run serially by default.

Arguably this is more maintainable than other design choices - there is one place that needs to be updated (the test) rather than all the scripts which run the test.

Paweł