Build and run just a single test on the try bots?

[Drew Wilson]

Hi all,

Sorry for the naive question, but my chromium-fu is a bit rusty.

Is it still possible to just build and execute a single browser test on the try bots? It looks like I can use "git cl try -b <bot>" to run the tests on a single bot, but if all I want is to execute a single browser test, running the full suite of tests takes way too long.

I'm trying to debug an error that only happens on the trybots, not locally (most likely some race condition).

-atw

[Jochen Eisinger]

Not possible anymore afaik :/

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAArhhisH-2Yk_gLnLAkSssnDrpRj-8Wx%2Bp-BNt0KiweZfqD4fw%40mail.gmail.com.

[Dirk Pranke]

You can do this, but it is convoluted and we should (and can) make it a lot easier.

Let's say you wanted to run ExtensionWebRequestApiTest.WebRequestUnloadImmediately for renderer_side_navigation_browser_tests, which is currently failing on the "Linux Tests" bot on the chromium.linux waterfall:

https://ci.chromium.org/buildbot/chromium.linux/Linux%20Tests/63487

First, you need to figure out which trybot to use. The trybots are mostly configured to "mirror" waterfall bots, and that mapping is listed in trybots.py:

https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/trybots.py

Searching for "Linux Tests" in that file, you'll see it maps to linux_chromium_rel_ng.

Second, the list of tests that are run are configured via the //testing/buildbot/*.json files in the src repo. There is one file per master, and you want the chromium.linux.json file.

Search in that file for the "Linux Tests" bot:

https://cs.chromium.org/search/?q=chromium.linux.json+"linux+tests"

and then scroll down to find the "renderer_side_navigation_browser_tests" entry:

https://cs.chromium.org/chromium/src/testing/buildbot/chromium.linux.json?l=3544

You need to modify that entry, and add argument to run just that file: `--gtest_filter=ExtensionWebRequestApiTest.WebRequestUnloadImmediately`.

and add that as an entry to the "args" value, right above it. If you wanted to run a list files, you can also use a "filter file"; there's a convenient example of that right below the lines in that file.

You're now almost there. There's one last step, because the way the "analyze" step on the trybots is implemented, it doesn't know how to deal with changes to the //testing/buildbot/*.json files, so it gives up and reruns all of the tests. So, your single browser test will be run, but so will all of the other test steps. In order to get around that, you need to remove the json files from the analyze whitelist:

https://cs.chromium.org/chromium/src/testing/buildbot/trybot_analyze_config.json?l=25

Wrap all that up and you get the following CL:

https://crrev.com/c/723721

You can then trigger the try job you want via the UI, or, if you're on the branch that you uploaded that CL from:

`git cl try -m master.tryserver.chromium.linux -b linux_chromium_rel_ng`.

Which I've just done, so we can see if I got all this right or not :).

-- Dirk

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CALjhuidu%2B13E4PuPn7xGt0zLwbCTdEbnR7WgpH2D%2BauqTzJpbQ%40mail.gmail.com.

[Dirk Pranke]

On Tue, Oct 17, 2017 at 9:37 AM, Dirk Pranke <dpr...@chromium.org> wrote:

You can do this, but it is convoluted and we should (and can) make it a lot easier.

Let's say you wanted to run ExtensionWebRequestApiTest.WebRequestUnloadImmediately for renderer_side_navigation_browser_tests, which is currently failing on the "Linux Tests" bot on the chromium.linux waterfall:

https://ci.chromium.org/buildbot/chromium.linux/Linux%20Tests/63487

First, you need to figure out which trybot to use. The trybots are mostly configured to "mirror" waterfall bots, and that mapping is listed in trybots.py:

https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/trybots.py

Searching for "Linux Tests" in that file, you'll see it maps to linux_chromium_rel_ng.

Second, the list of tests that are run are configured via the //testing/buildbot/*.json files in the src repo. There is one file per master, and you want the chromium.linux.json file.

Search in that file for the "Linux Tests" bot:

https://cs.chromium.org/search/?q=chromium.linux.json+"linux+tests"

and then scroll down to find the "renderer_side_navigation_browser_tests" entry:

https://cs.chromium.org/chromium/src/testing/buildbot/chromium.linux.json?l=3544

You need to modify that entry, and add argument to run just that file: `--gtest_filter=ExtensionWebRequestApiTest.WebRequestUnloadImmediately`.

and add that as an entry to the "args" value, right above it. If you wanted to run a list files, you can also use a "filter file"; there's a convenient example of that right below the lines in that file.

You're now almost there. There's one last step, because the way the "analyze" step on the trybots is implemented, it doesn't know how to deal with changes to the //testing/buildbot/*.json files, so it gives up and reruns all of the tests. So, your single browser test will be run, but so will all of the other test steps. In order to get around that, you need to remove the json files from the analyze whitelist:

https://cs.chromium.org/chromium/src/testing/buildbot/trybot_analyze_config.json?l=25

Wrap all that up and you get the following CL:

https://crrev.com/c/723721

You can then trigger the try job you want via the UI, or, if you're on the branch that you uploaded that CL from:

`git cl try -m master.tryserver.chromium.linux -b linux_chromium_rel_ng`.

Which I've just done, so we can see if I got all this right or not :).

And ... I didn't :).

There were two more steps. First, because analyze was still enabled, my CL didn't actually modify any source files, and so at first nothing was run. You can make a dummy source change to force something to happen, or you can leave the whitelist entry in (to bypass analyze).

Second, when you do modify files, it re-runs not only that test, but any other test that might be affected; in my case, that re-ran three other variants of the test.

To avoid that, you can modify the *.json entry to remove all of the other test steps.

I'll upload all of this to something in //docs for future reference.

-- Dirk

[Ryan Hamilton]

​I have wanted to do this a million times but didn't know how. Thanks for writing this up!! I can't wait to see it in //docs!​

[Drew Wilson]

Thanks, Dirk! Appreciate you digging into this.

[Paweł Hajdan, Jr.]

+infra-dev,maruel

You could also experiment with swarming, i.e. just trigger the task for binary with arguments you want, like --gtest_filter.

I wish this was easier, but it's a large effort.

Paweł

[Scott Graham]

Related to this, I recently added a script in tools/fuchsia/run-swarmed.py that's a wrapper around swarming. I've been using it to work on deflaking tests (unfortunately running this way still requires @google.com credentials when running directly rather than through the tryserver). Running directly on swarming like this is much quicker and resource-efficient than doing the same thing with tryjobs.

This isn't quite what was originally asked for as it only does the running part, not the build like a tryserver does. But if you can do the build locally, then you can then use the script to run on bots. And it accepts --gtest_filter.

(It's also Fuchsia-only at the moment, but it'd just need a switch to adjust the swarming dimensions if someone wanted to adapt it for other platforms.)

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAATLsPa2Rn2LqxP7q3oZ%3Drnxbb8p0RPHm1KtoxPuP%3DJCpb8bGQ%40mail.gmail.com.

[Dirk Pranke]

`mb run` can be used to run tests locally in an isolate, and I've long meant to extend it to run things under swarming as well. We should look and see if it makes sense to merge your run-swarmed.py script into it.

Ultimately I want the build to be generating wrapper scripts so that we could have an `//out/Release/bin/run_base_unitttests --swarmed` that would just do the right thing.

-- Dirk

[Nico Weber]

On Thu, Oct 19, 2017 at 4:51 PM, Scott Graham <sco...@chromium.org> wrote:

Related to this, I recently added a script in tools/fuchsia/run-swarmed.py that's a wrapper around swarming. I've been using it to work on deflaking tests (unfortunately running this way still requires @google.com credentials when running directly rather than through the tryserver). Running directly on swarming like this is much quicker and resource-efficient than doing the same thing with tryjobs.

I'm doing the same thing your script is doing at https://cs.chromium.org/chromium/src/docs/win_cross.md?type=cs&q=win_cross.md&sq=package:chromium&l=34 – I couldn't find a way to pass --gtest_filter to swarming. Were you able to do that? How?

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CANHK6RaFO%2B0AxGq0dR%2B%3DG_i3JfvHJAfSB9KmnVUNoHO%2BGsnCUA%40mail.gmail.com.

[Scott Graham]

swarming.py <args-to-swarming> -- --args-to-target-process-like--gtest_filter

[Julie Parent]

Mirroring what Dirk said earlier, I really want us to make this easier.  If I had $1 for every time a developer has asked for this, I could ... uh, almost hire an engineer to work on it? :)

The help folks have given here is useful, but I don't want us to lose track of the larger request.  Has anyone filed a request for this?  And, Dirk, can we get you on the hook for the "Thanks for writing this up!! I can't wait to see it in //docs!​" part :)

swarming.py <args-to-swarming> -- --args-to-target-process-like--gtest_filter

You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+unsubscribe@chromium.org.
To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CANHK6RYk-i6K4oeXBbF3xsbEYmjWKDxU06bL_78FvUQwjpC01g%40mail.gmail.com.

[Dirk Pranke]

Yes, I'm on the hook for the //docs part. I've also filed crbug.com/776611 for the larger request.

-- Dirk

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAPSmAARKKZ_byowpg9zcUzR9F8AHWKUD_3m82e922RN06QP2xg%40mail.gmail.com.