How to run telemetry_perf_unittests locally?

[Carlos Knippschild]

I have a change that causes crashes on a linux trybot when running steps telemetry_perf_unittests and telemetry_unittests. I need to debug them locally and I've been trying to do so for a while without success. Are there any instructions I can follow to do that?

[Sami Kyostila]

What kind of problems are you running into? Are you able to get the tests running with a command like:

$ tools/telemetry/run_tests --browser=release --chrome-root=. some.test.name.goes.here

or

$ tools/perf/run_tests --browser=release --chrome-root=. some.test.name.goes.here

- Sami

2015-11-24 11:04 GMT+00:00 Carlos Knippschild <car...@chromium.org>:

I have a change that causes crashes on a linux trybot when running steps telemetry_perf_unittests and telemetry_unittests. I need to debug them locally and I've been trying to do so for a while without success. Are there any instructions I can follow to do that?

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

[Yuta Kitamura]

I think the problem here is how to figure out that command line (the actual command line the bots are running).

When I ran into the similar situation in the past, I was also puzzled over it for some time. Maybe we could add simple scripts named telemetry_perf_unittests and telemetry_unittests in some directory?

[Carlos Knippschild]

Yuta is correct: the biggest issue is figuring out what command line to run. As those bots run tests using swarming I couldn't find myself what is the command line they use. It might be obvious for someone working in telemetry but not for me at least. It would be nice to have instructions available in a public page with a bit more context.

I did find both telemetry_perf_unittests.py and telemetry_unittests.py scripts in src/testing/scripts because I found this command line in the bot logs:

python -u /b/build/slave/android/build/src/testing/scripts/telemetry_perf_unittests.py --build-config-fs Release --use-src-side-runtest-py --paths /tmp/tmpdBcs_4.json --properties /tmp/tmpglPSdT.json run --output /tmp/tmp9PsHbY.json

But even so, I couldn't find the right configuration to run them locally. Maybe they are not even supposed to be used like that?

Now using the command line Sami provided I was able to run the tests I needed. I only had to change --browser to "exact" though (and then provide the path to it with --browser-executable) as using just "release" or "debug" didn't work for me. Also nice to know one can run several tests by naming them one after the other in the same command line.

[Daniel Cheng]

http://www.chromium.org/developers/telemetry/telemetry-unittests is the main documentation for these tests as far as I know.

[Carlos Knippschild]

Thanks Daniel. My fault not to find it then as the instructions are there.

[Scott Hess]

It would be useful if logs contained a minimized repro line appropriate for an individual contributor to execute locally.  In fact, it would additionally be useful if it contained a minimized config line, and a minimized build line.  Often the hump preventing me from working on a report is that I can't figure out even where to start, so I know I'm going to have to spend some time researching things to figure it out.  It's demotivating to think in terms of spending an hour or so reverse-engineering the test's environment to figure out how to isolate my failure, and then five minutes actually fixing the problem.

As an example of what I mean, something like:

~~~

Developer repro suggestion:

GYP_DEFINES="component=shared_library use_goma=1" gclient sync

ninja out/Debug unit_tests

./out/Debug/unit_tests --gtest_filter=MyTest.WhichFailed

~~~

-scott

[Carlos Knippschild]

+1 for this case and all other tests where this could be applied. This would help saving many engineering hours.

[Marc-Antoine Ruel]

Side note;

While it doesn't help when trying to build locally to reproduce, which was the problem here, you can download and run any isolated task as-is with python swarming.py reproduce. The exact command line to use is listed on each Swarming task page.

Thanks,

M-A

[Daniel Cheng]

What's the Swarming task page? =)

Daniel

[Marc-Antoine Ruel]

It's the "shard #N" link in the build step.

M-A

[Scott Hess]

To be clear, things that are obvious to people who are building and maintaining the bots are not necessarily obvious to the people being asked to fix problems.  There's a ton of information at every level, most of which is not relevant to the thing you're being told you need to research.  The output of the original poster's failure had thousands of lines of output, the build status page has hundreds of steps, etc.  As I mentioned earlier, the less poking around I have to do to even get started, the more likely it is that I'll get started immediately.

BTW, when I select a "Shard 0" link, my Chromium account gives me "403 Forbidden".  Perhaps it would work with my Google account, I don't know, but AFAICT there's no way to tell that page I want to give it another go with another account.

-scott

[Marc-Antoine Ruel]

Yes you have to use the @google.com account, opening it up is on my plate.

M-A